CS6320 – Performance more details
L. Grewe
1
System Architecture
Tier 1
Tier 2
Tier 3
DMS
Client
Web Server
Application
Server
Database
Server
Performance Desires and Approaches
• Improving performance and reliability to provide
– Higher throughput
– Lower latency (i.e., response time)
– Increase availability
• Some Approaches
– Scaling/Replication
• How performance, redundancy, and reliability are related to scalability
– Load balancing
– Web caching
3
Where to Apply Scalability
• To the network
• To individual servers
• Make sure the network has capacity
before scaling by adding servers
4
An example…but, first Hardware
Review
• Firewall
– Restricts traffic based on rules and can “protect” the internal network from
intruders
• Router
– Directs traffic to a destination based on the “best” path; can
communicate between subnets
• Switch
– Provides a fast connection between multiple servers on the same subnet
• Load Balancer
– Takes incoming requests for one “virtual” server and redirects them to
multiple “real” servers
Switch: Conencting More than 2 Machines
Case Study: Retail eBusiness
Internet
Data Circuit
This is the initial design
Router
PROBLEM: site is growing
and too many usersperformance is inadequate
Switch
Web
Server
Database
Server
7
Solution - Scaling
Internet
Data Circuit
Scaling through
Replication of systems
Web
Server
Web
Server
Web
Server
Router
Switch
Web
Server
Web
Server
Web
Server
Database Database
Server
Server
8
Internet
FIREWALL
ROUTER
LOAD
BALANCER
SWITCH
VLAN2
Firewall Router
Load
Balancer
Catalyst
VLAN2
WEB
SERVERS
SWITCH
VLAN3
APPLICATION
FIREWALL
SWITCH
VLAN4
DATABASE
SERVERS
Initial Redesign
Scaling mostly the web
servers.
Catalyst
VLAN3
Firewall
Problem: still have one
Entrance through firewall
for clients. A bottleneck
Catalyst
VLAN4
9
The Redesign Again:
Internet
Primary
FIREWALL
ROUTERS
SWITCHES
VLAN1
ry
ma n
Pri ectio
nn
o
C
Firewall Router
Primary
Red
u
Con ndant
nec
tion
Redundant
Firewall Router
Backup
Catalyst 1
VLAN1
Catalyst 2
VLAN1
LOAD
BALANCERS
Load
Balancer
Load
Balancer
SWITCHES
VLAN2
Catalyst 1
VLAN2
Catalyst 2
VLAN2
Last design: still bottleneck
Coming in on one path ….
Here we split into 2 “connected”
Paths.
WEB
SERVERS
SWITCHES
VLAN3
Catalyst 1
VLAN3
Catalyst 2
VLAN3
FIREWALLS
Primary FW
SWITCHES
VLAN4
DATABASE
SERVERS
Catalyst 1
VLAN4
Backup FW
Catalyst 2
VLAN4
10
Performance, Redundancy, and
Scalability
• Scale for performance
• But what about redundancy? Site going down.
11
How to get rid of Single
Points of Failure (SPOF):
Internet
a n t Co
nnectio
n
New York
Prim
n
d
Re
da
un
nt
e
nn
Co
ctio
n
Firewall Router
Backup
Redund
ect
io
Firewall Router
Primary
ection
nn
California
Conn
ary
Co
ry
Prima
Firewall Router
Primary
Firewall Router
Backup
Catalyst 1
VLAN1
Catalyst 2
VLAN1
Catalyst 1
VLAN1
Catalyst 2
VLAN1
Load
Balancer
Load
Balancer
Load
Balancer
Load
Balancer
Catalyst 1
VLAN2
Catalyst 2
VLAN2
Catalyst 1
VLAN2
Catalyst 2
VLAN2
Catalyst 1
VLAN3
Catalyst 2
VLAN3
Catalyst 1
VLAN3
Catalyst 2
VLAN3
Primary FW
Catalyst 1
VLAN4
Backup FW
Catalyst 2
VLAN4
Primary FW
Catalyst 1
VLAN4
Problem: Last design
if services to the single
geographical network
go down…site is down.
Answer: Replicate in
different geographical
locations
Backup FW
Catalyst 2
VLAN4
12
Scaling Servers: Out or Up
• Scale Out (Horizontal)..we saw this in previous
design
– Multiple servers
– Add more servers to scale
– Most commonly done with web servers
• Scale Up (Vertical)
– Fewer larger servers to add more internal resources
– Add more processors, memory, and disk space
– Most commonly done with database servers
13
Some Approaches to Scalability
• Approaches
– Farming
– Cloning
– RACS
– Partitioning
– RAPS
• Load balancing
• Web caching
14
Farming
This is about the
HW scaling
• Farm - the collection of all the servers,
applications, and data at a particular site.
– Farms have many specialized services (i.e.,
directory, security, http, mail, database, etc.)
15
Simple Web Farm
Cloning
This is about
Service / SW
replication
• A service can be cloned on many replica
nodes, each having the same software and
data.
• Cloning offers both scalability and availability.
– If one is overloaded, a load-balancing system can
be used to allocate the work among the
duplicates.
– If one fails, the other can continue to offer service.
17
Two Clone Design Styles
•Shared Nothing is simpler to implement and scales
IO bandwidth as the site grows.
•Shared Disc design is more economical for large or
update-intensive databases.
18
Reliable Array of Cloned Services (RACS)
• RACS (Reliable Array of Cloned Services)
– a collection of clones for a particular service
– shared-nothing RACS
• each clone duplicates the storage locally
• updates should be applied to all clone’ s storage
– shared-disk RACS (cluster)
• all the clones share a common storage manager
• storage server should be fault-tolerant
• subtle algorithms need to manage updates (cache
invalidation, lock managers, etc.)
19
Clones and RACS
• can be used for read-mostly applications with low
consistency requirements.
– i.e., Web servers, file servers, security servers…
• requirements of cloned services:
– automatic replication of software and data to new
clones
– automatic request routing to load balance the work
– route around failures
– recognize repaired and new nodes
20
Some definitions - Partitions and
Packs
•Data Objects (mailboxes, database records, business
objects,…) are partitioned among storage and server nodes.
•For availability, the storage elements may be served by a pack
of servers.
21
Partition
• grows a service by
– duplicating the hardware and software
– dividing the data among the nodes (by object), e.g.,
mail servers by mailboxes
• should be transparent to the application
– requests to a partitioned service are routed to the
partition with the relevant data
• does not improve availability
– the data is stored in only one place
– partitions are implemented as a pack of two or more
nodes that provide access to the storage
22
Taxonomy of Scaleability Designs
23
Reliable Array of Partitioned Services
RAPS
• RAPS (Reliable Array of Partitioned Services)
– nodes that support a packed-partitioned service
– shared-nothing RAPS, shared-disk RAPS
• Update-intensive and large database
applications are better served by routing
requests to servers dedicated to serving a
partition of the data (RAPS).
24
Some Approaches to Scalability
• Approaches
– Farming
– Cloning
– RACS
– Partitioning
– RAPS
• Load balancing
• Web caching
25
Load Balancing / Sharing
26
Load Management
• Balancing loads (load balancer) can operate at
different OSI layers
– Round-robin DNS
– Layer-4 (Transport layer, e.g. TCP) switches
– Layer-7 (Application layer) switches
The 7 OSI (Open System Interconnection) Layers
(a model of a network)
Load Balancing Strategies
• Flat architecture
– DNS rotation, switch based, MagicRouter
• Hierarchical architecture
• Locality-Aware Request Distribution
29
DNS Rotation - Round Robin Cluster
30
Flat Architecture - DNS Rotation
•
DNS rotates IP addresses of a Web site
–
•
Pros:
–
•
expensive, inefficient
Switching products
–
–
–
–
•
Client-side IP caching: load imbalance, connection to down node
Hot-standby machine (failover)
–
•
A simple clustering strategy
Cons:
–
•
treat all nodes equally
Cisco, Foundry Networks, and F5Labs
Cluster servers by one IP
Distribute workload (load balancing)
Failure detection
Problem
–
Not sufficient for dynamic content
31
Load
Balance
Idea 2:
Switchbased
Cluster
32
Flat Architecture - Switch Based
• Switching products
– Cluster servers by one IP
– Distribute workload (load balancing)
• i.e. round-robin
– Failure detection
– Cisco, Foundry Networks, and F5Labs
• Problem
– Not sufficient for dynamic content
33
Problems with DNS or Switch Load
Balancing
• Problems
– Not sufficient for dynamic content
– Adding/Removing nodes can be involved
• Manual configuration required
– limited load balancing in switch
– Simple algorithms do not consider current loads
34
Load Sharing Strategies
• Flat architecture
– DNS rotation, switch based, MagicRouter
• Hierarchical architecture
• Locality-Aware Request Distribution
35
Hierarchical Architecture
• Master/slave architecture
• Two levels
– Level I
• Master: static and dynamic content
– Level II
• Slave: only dynamic
36
Hierarchical Architecture
M/S Architecture
37
Hierarchical Architecture
38
Hierarchical Architecture
• Benefits
– Better failover support
• Master restarts job if a slave fails
– Separate dynamic and static content
• resource intensive jobs (CGI scripts) runs by slave
• Master can return static results quickly
39
Locality-Aware Request Distribution
• Content-based distribution
– Improved hit rates
– Increased secondary storage
– Specialized back end servers
• Architecture
– Front-end
• distributes request
– Back-end
• process request
40
Load Sharing Strategies
• Flat architecture
– DNS rotation, switch based, MagicRouter
• Hierarchical architecture
• Locality-Aware Request Distribution
41
Locality-Aware Request Distribution
Naïve Strategy
42
Some Approaches to Scalability
• Approaches
– Farming
– Cloning
– RACS
– Partitioning
– RAPS
• Load balancing
• Web caching
43
Web Caching
44
Web Proxy
• Intermediate between clients and Web servers
• It is used to implement firewall
• To improve performance, proxy caching
Client (browser)
Web server
With Caching
45
Web Architecture
• Client (browser), Proxy, Web server
Web server
Firewall
Proxy
Client (browser)
46
Web Caching not only at Proxy
Servers
• Caching popular objects is one way to
improve Web performance.
• Web caching at clients, proxies, and Web server
Proxy
servers.
Client (browser)
47
Advantages of Web Caching
• Reduces bandwidth consumption (decrease
network traffic)
• Reduces access latency in the case of cache hit
• Reduces the workload of the Web server
• Enhances the robustness of the Web service
• Usage history collected by Proxy cache can be
used to determine the usage patterns and
allow the use of different cache replacement
and prefetching policies.
48
Disadvantages of Web Caching
• Stale data can be serviced due to the lack of
proper updating
• Latency may increase in the case of a cache
miss
• A single proxy cache is always a bottleneck.
• A single proxy is a single point of failure
• Client-side and proxy cache reduces the hits
on the original server.
49
Web Caching Issues
•
•
•
•
Cache replacement
Prefetching
Cache coherency
Dynamic data caching
50
Cache Replacement
• Characteristics of Web objects
– different size, accessing cost, access pattern.
• Traditional replacement policies do not work well
– LRU (Least Recently Used), LFU (Least Frequently
Used), FIFO (First In First Out), etc
• There are replacement policies for Web objects:
– key-based
– cost-based
51
Caching -Two Replacement Schemes
• Key-based replacement policies:
– Size: evicts the largest objects
– LRU-MIN: evicts the least recently used object among ones with largest
log(size)
– Lowest Latency First: evicts the object with the lowest download latency
• Cost-based replacement policies
– Cost function of factors such as last access time, cache entry time, transfer
time cost, and so on
– Least Normalized Cost Replacement: based on the access frequency, the
transfer time cost and the size.
– Server-assisted scheme: based on fetching cost, size, next request time, and
cache prices during request intervals.
52
Caching -Prefetching
• The benefit from caching is limited.
– Maximum cache hit rate - no more than 40-50%
– to increase hit rate, anticipate future document
requests and prefetch the documents in caches
• documents to prefetch
– considered as popular at servers
– predicted to be accessed by user soon, based on the
access pattern
• It can reduce client latency at the expense of
increasing the network traffic.
53
Cache Coherence
• Cache may provide users with stale documents.
• HTTP commands for cache coherence
– GET : retrieves a document given its URL
– Conditional GET: GET combined with the header IFModified-Since.
– Progma: no-cache : this header indicate that the
object be reloaded from the server.
– Last-Modified : returned with every GET message and
indicate the last modification time of the document.
• Two possible semantics
– Strong cache consistency
– Weak cache consistency
54
Strong cache consistency
• Client validation (polling-every-time)
– sends an IF-Modified-Since header with each access of
the resources
– server responses with a Not Modified message if the
resource does not change
• Server invalidation
– whenever a resource changes, the server sends
invalidation to all clients that potentially cached the
resource.
– Server should keep track of clients to use.
– Server may send invalidation to clients who are no
longer caching the resource.
55
Weak Cache Consistency
– Adaptive TTL (time-to-live)
• adjust a TTL based on a lifetime (age) - if a file has not been modified for a long
time, it tends to stay unchanged.
• This approach can be shown to keep the probability of stale documents within
reasonable bounds ( < 5%).
• Most proxy servers use this mechanism.
• No strong guarantee as to document staleness
– Piggyback Invalidation
• Piggyback Cache Validation (PCV) - whenever a client communicates with a server,
it piggybacks a list of cached, but potentially stale, resources from that server for
validation.
• Piggyback Server Invalidation (PSI) - a server piggybacks on a reply to a client, the
list of resources that have changed since the last access by the client.
• If access intervals are small, then the PSI is good. But, if the gaps are long, then the
PCV is good.
56
Dynamic Data Caching
• Non-cacheable data
– authenticated data, server dynamically generated data, etc.
– how to make more data cacheable
– how to reduce the latency to access non-cacheable data
• Active Cache
– allows servers to supply cache applets to be attached with documents.
– the cache applets are invoked upon cache hits to finish necessary processing
without contacting the server.
– bandwidth savings at the expense of CPU costs
– due to significant CPU overhead, user access latencies are much larger than
without caching dynamic objects.
57
Dynamic Data Caching
• Web server accelerator
– resides in front of one or more Web servers
– provides an API which allows applications to
explicitly add, delete, and update cached data.
– The API allows static/dynamic data to be cached.
– An example - the official Web site for the Olympic
Winter Games
• whenever new content became available, updated Web
reflecting these changes were made available quickly.
• Data Update Propagation (DUP, IBM Watson) is used for
improving performance.
58
Dynamic Data Caching
• Data Update Propagation (DUP)
– maintains data dependence information between
cached objects and the underlying data which affect
their values
– upon any change to underlying data, determines
which cached objects are affected by the change.
– Such affected cached objects are then either
invalidated or updated.
– With DUP, about 100% cache hit rate at the 1998
Olympic Winter Games official Web site.
– Without DUP, 80% cache hit rate at the 1996 Olympic
Games official Web site.
59
Towards Large Scale system ….and
need for clusering
• Large scale systems (think Yahoo!, YouTube,
Ebay, Amazon, Google)…
60
One Large Scale Need – High
Availability
• High availability is a major driving requirement behind largescale system design. Basically, means the system is available
(and responding) a high percentage of the time.
– Uptime: typically measured in nines, and traditional infrastructure
systems such as the phone system aim for four or five nines (“four
nines” implies 0.9999 uptime, or less than 60 seconds of downtime per
week).
61
High Availability – how to measure
– Meantime-between-failure (MTBF)
– Mean-time-to-repair (MTTR)
– uptime = (MTBF – MTTR)/MTBF
– yield = queries completed/queries offered
– harvest = data available/complete data
– DQ Principle:
Data per query × queries per second →constant (total data delivered)
•
•
•
•
•
System level physical bottleneck
Total I/O bandwidth (disk or network)
Optimization goal is to minimize the
utilization of the bottleneck resource
Fault tolerance: trade-off between D and Q
Graceful Degradation is a goal
Using High Availability metrics to
compare Replication vs.
Partitioning
Replication of data in 2 nodes
• – 1 failure: 100% harvest (D), 50% yield (Q)
Partition of data in 2 nodes
• – 1 failure: 50% harvest (D), 100% yield (Q)
63
Cluster Example
Smaller to
mid-sized
Cluster
Example.
Large
examples
like
Amazon
have in
the
thousands
nodes
Some Tips
•
Get the basics right. Start with a professional data center and layer-7 switches, and use
symmetry to simplify analysis and management.
•
Decide on your availability metrics. Everyone should agree on the goals and how to measure
them daily. Remember that harvest and yield are more useful than just uptime.
•
Focus on MTTR at least as much as MTBF. Repair time is easier to affect for an evolving
system and has just as much impact.
•
Understand load redirection during faults. Data replication is insufficient for preserving
uptime under faults; you also need excess DQ.
•
Graceful degradation is a critical part of a high-availability strategy. Intelligent admission
control and dynamic database reduction are the key tools for implementing the strategy.
•
Use DQ analysis on all upgrades. Evaluate all proposed upgrades ahead of time, and do
capacity planning.
•
Automate upgrades as much as possible. Develop a mostly automatic upgrade method, such
as rolling upgrades. Using a staging area will reduce downtime, but be sure to have a fast,
simple way to revert to the old version.