An Analysis of Facebook
photo Caching
Qi Huang, Ken Birman, Robbert van Renesse, Wyatt Lloyd,
Sanjeev Kumar,Harry C. Li
Cornell Univ, Princeton Univ, Facebook Inc
SOSP 2013
Content
Introduction
Facebook‘s Photo-Serving stack
Method
Potential Improvement
Conclusion
Critics
2
Introduction
Upload the 250 billions of Photo
and accessed client
1)
2)
3)
How much of the access traffic in server?
How request travel all photo-serving stack?
How different cache size and eviction algorithm?
3
Facebook‘s Photo-Serving stack
Local Photo-Serving stack
Client
Browser
Cache
Local
Fetch
4
Facebook‘s Photo-Serving stack
Geo-distributed Edge Cache (FIFO)
Client
PoP
Browser
Cache
(Millions)
Edge
Cache
(Nines)
5
Facebook‘s Photo-Serving stack
Single Global Origin Cache (FIFO)
Client
PoP
Browser
Cache
(Millions)
Edge
Cache
(Nines)
Data Center
Origin
Cache
(Four)
Hash(url)
6
Facebook‘s Photo-Serving stack
Haystack
Client
PoP
Browser
Cache
(Millions)
Edge
Cache
(Nines)
Data Center
Origin
Cache
Backend
(Haystack)
(Four)
7
Facebook‘s Photo-Serving stack
Organization of Photo-Serving stack
• Browser cache
- First cache layer and co-located with the client
- Using in-memory hash table
- Using LRU(Least recently used) eviction
• Edge cache
- Second cache layer and spreaded across US
- Using in-memory hash table and hold metadata
- Using FIFO(First In First Out) replacement policy
- Main Goal : traffic sheltering and bandwidth reduction
· Traffic sheltering : prioritization drive a number of decisions
8
Facebook‘s Photo-Serving stack
Organization of Photo-Serving stack
• Origin cache
- Thrid cache layer and co-located with backend
- Using hash mapping based on the unique id of the photo
- Using FIFO replacement policy
• Haystack
- Lowest level of the photo serving stack
- Using a compact blob(binary large object) and stored images
9
Facebook‘s Photo-Serving stack
Photo transformation
• Can serve many flatform
- such as mobile, desktop, laptop
• Resizer in backend and caching layer
- Resized and cropped photos saved Backend haystack machines
- Single cache has many size of same photo
- If Akamai CDN(Contents Delivery Network)reqest, dosen’t store
storage
10
Methodology
Trace Collection
• Objective
- Collecting a representative sample that could permits correlation of
events related to the same request
Client
PoP
Browser
Cache
Edge
Cache
Data Center
Origin
Cache
Backend
(Haystack)
Instrumentation Scope
11
Methodology
Methodology
• Client
- Changed code in facebook using javascript
· If user select photo in facebook browser load specific photo in sampling
· Periodically javascript uploads which is record
· Aggeregated result from multiple clients by webserver and report
• Edge cache/Origin cache
-
Report all situation in cache
Trivial problem
• This approach would be disruptive to the existing Facebook base
code
-
Give unique request ID
-
This information travel along the stack
12
Methodology
Sampling strategies
• Request-based
- Sampling requests randomly
- Bias on popular content
• Objected-based
- Focused on some subset of photo by deterministic test on photoId
- Fair coverage of unpopular photos
- Cross stack analysis
13
Workload
Analysis Objectives
•
•
•
•
Traffic sheltering effects of caches
Photo popularity distribution
Geographic traffic distribution
Idea which can make the cache better
- Impact of sizes & algorithm
14
Workload
Traffic Sheltering
Client
Data Center
PoP
Browser
Cache
Edge
Cache
Origin
Cache
R
Backend
(Haystack)
77.2M
65.5%
26.6M
58.0%
11.2M
7.6M
31.8%
Traffic Share 65.5%
20.0%
4.6%
9.9%
15
Workload
Popularity Distribution
• Popularity
• Rank shift
- Most popular object(3th-10th) dropped out
- More popular object(10th-100th) drop to 1000th-10000th
16
Workload
Hit Ratio
• Insight of distribution for distinct photo blobs
17
Geographic Traffic Distribution
Client to Edge cache traffic
18
Cross-Region Traffic at Backend
Cross-Region Traffic at Backend
• Misdirected resizing traffic
- Didn’t copy adequate number of backup copy
• Failed local fetch
Migration
19
Potential Improvement
Browser Cache
• What-if question : “what the client browser hit ratios would have
been with an infinite cache size?”
20
Potential Improvement
Edge Cache
• Test on San Jose Edge Cache
• Cache size
• algorithms
21
Potential Improvement
Origin Cache
• Cache size and algorithm
22
Social-Network Analysis
Content Age Effects
• Content Age
- Age of content that request photo upload in 24hours
- Exclud profile photo
23
Social-Network Analysis
Social Effects
• Effect of popularity
• Hit rate dependent on followers and many friends
24
Conclusion
Conclusion
•
•
•
•
Instrumented th entire Face book photo-serving stack
Traced representative of Facebook’s full workload
Workload include pattern, distribution, geographic system
Tested the option of cache that change algorithm
25
Critics
Using ‘intuitionally’ too much
26