Agenda




Introductions 10m
Cavisson Intro 10m
Product Presentation 30m
Next steps and Q&A 10m
Cavisson Systems, Inc

What we do?

For web based applications, Our Products
are aimed at




Optimizing User Experience
Perform Performance analysis
Provide Capacity Planning
US Patents pending technology
Cavisson Systems, Inc

Sample Customers





Wells Fargo bank, One of the top 5 global banks
Macys: US top retailer with over US$ 25 Billion in
annual sales
Oracle Corporation: Global software giant
Renesas: Japanese electronic component
conglomerate
A10 networks: US based Bleeding edge Network
component provider
Cavisson Systems, Inc

Who we are?





Leading edge technology provider
HQ in California, USA
Sales office in New York, USA
Development office in Noida, India
Core team



From top colleges (IIT’s, UC Berkley, Stanford)
History of building industry success products (CDOT,
India; AT&T; BEA Systems)
Solid experience with all aspects of web technology
Cavisson Systems, Inc

What’s unique about us!

We provide unique simulation solutions that
replicate the production environment in the lab.
Thus



Provide user experience measure
Optimize the applications for optimal resource utilization
under real-world usage
Help solve real-world production issues by replicating
them in the lab
Issues with emerging new esolutions






Failure to Plan is planning to fail
Emerging e-solution just focus on development
Little focus on deployment!
Established e-solutions understands the need of
deployment planning and hence load testing
In western world, Almost all enterprise class web
applications load tested
In India – still emerging


Much publicized election commission site failed during recent
Lok Sabha elections.
Because high visibility, became major news headline
Cavisson Systems, Inc

Because of unique capabilities


Customers like Macys switched from legacy
products to cavissson
IBM & Oracle sidestepped their own tools to
use cavisson solution on some their projects
Typical Web Application
Internet
Web Users
Web Application
Multi-tier
Servers
Web Application Load &
Performance Testing
Internet
Web Users
Emulated by NetStorm
Web Application
Multi-tier
Servers
Web Application Load &
Performance Testing
Emulated by NetStorm
Emulates
Internet &
Web Users
Web Application
Multi-tier
Servers
Why NetStorm & NetOcean?




High performance Real-world emulation
Emulate faithfully deployment
environment in the Lab
Provide Real-life view of web system
Capacity & performance
Extremely powerful – avoid network
and server cluttering
Geographically Distributed Users
With Diverse Network Access
Internet
A New York
Web User
with DSL
modem
Web Application
hosted in San Jose
A Typical Web transaction
Internet
New York
Web User
points
browser to
a web site
hosted in
San Jose
Web Application
hosted in San Jose
A Typical Web transaction
Browser Opens several
TCP connections to
download a page
Web Application
hosted in San Jose
Internet
A Typical Web transaction
Internet
All Parallel
connections
share same
modem
bandwidth
Web Application
hosted in San Jose
A Typical Web transaction
All Network Communication
faces SJ-NY WAN path
Internet
Web Application
hosted in San Jose
A Typical Web transaction
Internet
Server Gets request in
Internet Random pattern
Web Application
hosted in San Jose
A Typical Web transaction
Server faces Web User
Community behavior –
Reloads on slow, sudden
burst
Internet
Web Application
hosted in San Jose
A Typical Web transaction
Internet
Web Application
All discussed elements play vital role in
end-user response time & the server load hosted in San Jose
NetStorm:
Emulates all web elements
NetStorm provides Real world view of Enduser performance & Web System Load
NetStorm Appliance
Emulates
Internet &
Web Users
Web Application
Multi-tier
Servers
NetStorm: Internet in a box!




Emulates geographically distributed web
users
Emulates network latency, jitter, packet
drop and bandwidth impairments
Unique capability to emulate user
modem
Internet traffic emulation
Lab View Vs Real-World View
Things That
Matter
Response
Time
Lab View
(As Noted by
Legacy tools:
Load Runner )
1 Second
Real-World View
(As noted by Real users
and NetStorm)
Why Different?
6 Sec (For Broadband
Users averaged across
USA)
14 Sec (For Dial-up users
averaged across USA)
Lacking simulation of
Internet, Traffic pattern, User
Community (Network Access,
Location, browser and
behavior)
Server service
time optimized by
50%
User Response optimized
by 8%
Trying to optimize a
component without
understating big picture
Server service
time optimized by
3%
User Response optimized
by 30%
Overall response depends
upon a complex mix of
variables
System
capacity
60% busy @ 500
session/Hr
100% busy @ 500
session/Hr
Network Concurrency is over
100X in real word
Perceived
Problem
DB Server major
cause
Number of threads at App
Server Major cause
Focusing only on small subset of real-world setup
Performance
Optimization
Overall View of System
Performance including SUT
Co-Relate Server Performance
with End User Response
Page Download Response
Time for Geographic users
Would the response time for
Geographic users meet SLA?
Page download time drilldown
NetStorm: More for Le$$
Features
Testing Focus
Load generation (URL hits/sec)
Web Page Analysis
Browser emulation
SUT/DUT Stats & Co-relation
Internet Emulation
Internet Traffic pattern
Web User behaviour
Load Runner Smartbits
Applicaion
1000
Yes
Yes
Yes
No
No
No
Network
200,000
No
No
No
No
No
No
NetStorm
Web user experience
300,000
Yes
Yes
Yes
Yes
Yes
Yes
Why NetStorm & NetOcean?




Designed for Real-World load modeling of Enterprise scale in a small
hardware footprint (order of magnitude small compared to Legacy
systems such as Load Runner)
Real-world User modeling for accurate production system predictions
Provides Real-world view of system performance as opposed Lab-View
of system performance provided by Legacy Systems
Owing to its deep understanding and simulation of Network and
Application layers, NetStorm has



Solved real-life performance and production problems in large enterprises
environments (Such as Macys, Wells Fargo, Oracle, National City Bank,
Renesas,A10 Networks…).
Replaced Network or Application centric tools such as IXIA, Load Runner as
they fall short in identifying the root cause
Companies like IBM Global services and Oracle choose to use NetStorm,
sidestepping their own Load Generation tools
Case Study: Wells Fargo Bank







Planned to Deploy Load Balancer (SLB)
Tested in the lab with legacy solutions
Result suggested no change in performance
Load balancer deployed
Monitoring suggested increase in response
time
Rested – No clue about bad performance
Tested with NetStorm



NetStorm simulated the Real-world environment
Displayed higher response with SLB
Identified problem
Case Study: Macys

2007 Disaster – Session DB Growth
 Holiday season nightmare
 Webload could not predict
 Band-Aid solutions
 LoadRunner bought, no new prediction
 NetStorm identified root cause




Multiple WebSphere session getting created in single
user session
Owed to cookie not set for new users and thus creating
new session on user navigations
Concurrent user model employed by Load Runner could
not maintain arrival rate and thus the pace of session
creations
Daily Session count creations now down 10 folds for
same user sessions
Case Study: Macys

Holiday stability Preparations

Many of the past instability related to OutOfMemory JVM

Effort setup to make memory enhancements

IBM’s Rational Load tester was employed from IBM Labs to
create production sized load – Did not dent the system

LoadRunner 8000 VU employed with 18 Load generators
from IBM labs.



Did not cause any serious impact on JVM memory
Predict the new release with memory optimizations worse in
performance
Employed NetStorm




Single Hardware box
Created OutofMemory condition
Predicted - better heap utilization with new release and no
worse performance with new release
After deployment, Production stats vindicate NetStorm
prediction
Case Study: Macys

Server monitors at work!

NetStorm collects server signatures




Helped pin-point the trouble to changes in server
configuration when Webspehere upgrade resulted in
increased session count.
NetStorm’s JAVA GC monitor helped establish the
stable configuration
Helped identify the root cause of Enterprise service
bus seen in production. NetStorm Monitors pin-pointed
the problem to server memory leak.
Pin-pointed the root cause of server crash during
Webspehere upgrade. Found a file descriptor leak.
Identified the exact file name and process causing the
leak.
Case Study: Macys

Solid time savings

compact hardware foot-print (of the order of 1:20), test
setup extremely reliable. Earlier used Load testing solution
required reboot quite often and needed a lot of attention
because of so many moving parts. NetStorm not rebooted
even once since it was installed more than a year back.

Test Comparison extremely fast. With Earlier load
testing solutions, Test comparison was almost never
employed because of long time for test comparison.
NetStorm does it in a a couple of minutes using in-memory
database and hashes as opposed to what took several hours
to dissect and compare tests.

NetStorm’s test suite automation used effectively to tune
Java Virtual Machine (JVM) Performance. Tuning exercises
were never conducted effectively earlier because of too
much human time involvement.
Summary

Credible Stress Environment Load Generation





Why NetStorm?




Sized to replicate & stress Production System load
Ability to replicate geographically distributed Users
Ability to emulate Internet community behavior
Minimize moving parts to increase test reliability and cutting corners
Designed specifically to deal with Web Based systems, built on years of
research
Meets the goals & Objectives of Load Testing, Performance analysis &
Capacity estimation
Extremely powerful to simulate 100s of thousands real web users using single
hardware box
Solution of choice!



Solved real-life performance and production problems in large enterprises
environments (Such as Macys, Wells Fargo, Oracle, National City Bank,
Renesas, A10 Networks…).
Replaced Network or Application centric tools such as IXIA, Load Runner as
they fall short in identifying the root cause
Companies like IBM Global services and Oracle choose to use NetStorm,
sidestepping their own Load Generation tools
Next Steps

Websites

http://www.mplun.net/
https://mpeprocurement.com/

http://www.mrignayani.com/






for e-tenders
for shop on-line sales
Are there other websites under MP LUN?
Who maintains (makes changes/updates) to these sites? MP LUN or
some contracting company?
What kind of servers are used for hosting/running these websites?
Where are production server(s) hosted? Who maintain and monitors
these servers?
What are the Servers for testing and where they are located? And how
test server configuration compares? Are they similar to one used for
production system?
Web site traffic details in hits /day? What is maximum hit rate observed
on these sites?
Next Steps

Websites

http://www.mplun.net/
https://mpeprocurement.com/

http://www.mrignayani.com/






for e-tenders
for shop on-line sales
Are there other websites under MP LUN?
Who maintains (makes changes/updates) to these sites? MP LUN or
some contracting company?
What kind of servers are used for hosting/running these websites?
Where are production server(s) hosted? Who maintain and monitors
these servers?
What are the Servers for testing and where they are located? And how
test server configuration compares? Are they similar to one used for
production system?
Web site traffic details in hits /day? What is maximum hit rate observed
on these sites?
Next Steps

Available Load test tools
Features
Example
Typical packaging
Load generation
Application knowledge
Web user knowledge
Network knowledge
Server Stats Co-relation
Test Application server farm
Application focused Tools
Network focused tools
Mercury Load Runner
Software
Low: ~1000 URL hits/sec
High: Understands web age
High: Understands web browser
Low: No network simulation
Yes
Not available
Spirent Avalanche
Appliance
High;~ 100,000 URL hits/sec
Low: understands a URL hit
Low: Understands TCP connections
Medium: Provides network impairments
No
Available
NetStorm combines the strengths of both catagories
NetStorm & NetOcean
Web-based Application Load
Testing Solution
Cavisson Systems Inc.
Cavisson Proprietary and Confidential
Real-Life Network Cache
R1
R2
Network Cache
Origin server
Internet
Network cache usually serve
objects faster because of
physical proximity to users.
R1 > R2
Network Cache:
Lab Test Setup
R1
Internet Emulation
R2
NetStorm emulates Geographically distributed users with
Origin server at fixed location and Network cache at network
edge for all users
NetOcean acting as Network cache
Origin server
LAB
Web Application Load &
Performance Testing
Origin Servers
Emulated by NetStorm
Emulates
Internet &
Web Users
Network Cache Server
emulated by NetStorm
Case study: Wells Fargo Bank
Online Banking Authentication
Decides to use Load Balancer
 Pre-production tests used legacy load
testing tools, saw no difference
Before
After

Case study:Wells Fargo Bank
External monitoring Complains!


Response time gone up: 5.6 -> 6.2 sec
Mostly extra time gone in SSL
SLB Deployed
Case study:Wells Fargo Bank
NetStorm used for Analysis!
Metric
Direct
With SLB
Sign-On Load
1.212
1.29
Sign-On
3.601
3.875
Sign-Off
0.836
1.015
Total Sign On
5.649
6.18
New SSL Session/Sec
4.5
73.6
SSL Reuse Request/Sec
95.9
95.9
SSL Reused/Sec
95.9
26.7
Connections/Sec
100
100
Root Cause
No SLB Case
C1
WS1
C2
WS2
C3
C4
WS3
Web
Clients
Web Server Farm
Root Cause
No SLB Case
C1
WS1
C2
WS2
C3
1.
C1 resolves online.wellsfargo.com (WF) to WS1 IP Address
C4
WS3
Web
Clients
Web Server Farm
Root Cause
No SLB Case
C1
WS1
C2
WS2
C3
1.
C1 Resolves online.wellsfargo.com (WF) to WS1 IP Address
2.
Client C1 Makes TCP connection to WS1.
C4
WS3
Web
Clients
Web Server Farm
Root Cause
C1-WS1
SSID = 100
C1-WS1
SSID = 100
No F5 Case
C1
WS1
C2
WS2
C3
C4
Web
Clients
1.
C1 resolves online.wellsfargo.com (WF) to WS1 IP Address
2.
Client C1 Makes TCP connection to WS1.
3.
SSL handshake between Client C1 & WS1. Both have reference to
newly created SSL session ID = 100 (ID assigned by WS1)
WS3
Web Server Farm
Root Cause
C1-WS1
SSID = 100
C1-WS1
SSID = 100
No F5 Case
C1
WS1
C2
WS2
C3
C4
1.
Web Client C1 Resolves online.wellsfargo.com to WS1 IP Address
2.
Client C1 Makes TCP connection to WS1.
3.
SSL handshake between Client C1 t & WS1. Both have reference to
newly created SSL session ID = 100 (ID assigned by WS1)
4.
Web
Clients
WS3
Client opens another connection in same web session to
online.wellsfargo.com (DNS cached resolution points to WS1).
Web Server Farm
Root Cause
C1-WS1
SSID = 100
C1-WS1
SSID = 100
No F5 Case
C1
WS1
C2
WS2
C3
C4
Web
Clients
1.
C1 resolves online.wellsfargo.com (WF) to WS1 IP Address
2.
Client C1 Makes TCP connection to WS1.
3.
SSL handshake between Client C1 t & WS1. Both have reference to
newly created SSL session ID = 100 (ID assigned by WS1)
4.
Client opens another connection in same web session to
online.wellsfargo.com (DNS cached resolution points to WS1).
5.
Client request to reuse session ID 100 (pre-negotiated SSL). WS1
is happy to reuse session ID 100
WS3
Web Server Farm
Root Cause
With SLB
C1
WS1
C2
WS2
1.
C3
C1 Resolves online.wellsfargo.com (WF) to SLB IP Address
C4
WS3
We Clients
Web Server Farm
Root Cause
With SLB
C1
WS1
C2
WS2
C3
1.
C1 Resolves online.wellsfargo.com (WF) to SLB IP Address
2.
Client C1 makes TCP connection to SLB that is forwarded to WS1.
C4
WS3
We Clients
Web Server Farm
Root Cause
C1-WF
SSID=100
C1
C1-WF
SSID=100
With SLB
WS1
C2
WS2
C3
1.
Web Client C1 Resolves online.wellsfargo.com to SLB IP Address
2.
Client C1 Makes TCP connection to SLB that is forwarded to WS1.
3.
SSL handshake between Client C1 & WS1. Both have reference to
newly created SSL session ID = 100 (ID assigned by WS1)
C4
WS3
We Clients
Web Server Farm
Root Cause
With SLB
C1
WS1
C2
WS2
C3
C4
1.
Web Client C1 Resolves online.wellsfargo.com to SLB IP Address
2.
Client C1 Makes TCP connection to SLB that is forwarded to WS1.
3.
SSL handshake between Client C1 & WS1. Both have reference to
newly created SSL session ID = 100 (SSID assigned by WS1)
4.
Client Makes another request in same web session to SLB that is
forwarded to WS3 to load balance
WS3
We Clients
Web Server Farm
Root Cause
C1
C1-WF
SSID=200
C1-WF
SSID=100
With SLB
WS1
C2
WS2
C3
1.
C1 Resolves online.wellsfargo.com (WF) to SLB IP Address
2.
Client C1 Makes TCP connection to SLB that is forwarded to WS1.
3.
SSL handshake between Client C1 & WS1. Both have reference to
newly created SSL session ID = 100 (SSID assigned by WS1)
4.
Client Makes another request in same web session to SLB that is
forwarded to WS3 to load balance
5.
Client request to reuse 100 (pre-negotiated SSL). WS3 has no
knowledge of SSID = 100 and forces a new handshake with id 200
C4
Web Clients
C1-WF
SSID=200
WS3
Web Server Farm
Root Cause
C1
C1-WF
SSID=200
C1-WF
SSID=100
With SLB
WS1
C2
WS2
C3
1.
C1 Resolves online.wellsfargo.com (WF) to SLB IP Address
2.
Client C1 Makes TCP connection to SLB that is forwarded to WS1.
3.
SSL handshake between Client C1 & WS1. Both have reference to
newly created SSL session ID = 100 (SSID assigned by WS1)
4.
Client Makes another request in same web session to SLB that is
forwarded to WS2 to load balance
5.
Client request to reuse 100 (pre-negotiated SSL). WS3 has no
knowledge of SSID = 100 and forces a new handshake with id 200
6.
For a third new connection, Client will now request SSL reuse with
SSID=200. If the new connection happen to served by any other
web server than WS3, server will force new SSL handshake.
C4
Web Clients
C1-WF
SSID=200
WS3
Web Server Farm
Investigate options with NS:
Keep-Alive is the winner!
Metric
Direct
With SLB
SSL Sticky
Keep-Alive
One Teir
Sign-On Load
1.212
1.29
1.294
1.142
1.221
Sign-On
3.601
3.875
3.788
2.861
2.815
Sign-Off
0.836
1.015
0.84
0.426
0.363
Total Sign On
5.649
6.18
5.922
4.429
4.399
New SSL Session/Sec
4.5
73.6
4.5
22.4
6.7
SSL Reuse Request/Sec
95.9
95.9
95.9
24.6
22.3
SSL Reused/Sec
95.9
26.7
95.9
6.6
22.3
Connections/Sec
100
100
100
29
29
NetStorm at Wells:
60 Sec Keep-Alive wins!
Online Sign-on Response Time (All Objects)
8
7
6
5
4
3
2
1
0
Response Time
Direct NKA
F5 NKA
F5 15 Sec
KA
F5 30 Sec
KA
F5 60 Sec
KA
F5 90 Sec
KA
6.836
7.224
5.825
5.514
5.375
5.491