Wix Strategy Review

Prospero Media Storage
Managing 100TB of small files…
IGT –
Event
July 2011
Numbers
•
•
•
•
•
•
•
•
70TB used space
700 million files
200GB and 250,000 files uploaded every day
1200Mbps bandwidth throughput in peak
180TB of data is being served out monthly
3700 Hits per second in peak
40 storage node servers – 300TB raw space
$0.13 per GB
Motivation
• Web 2.0 content serving paradigm shift
– Too many files
• 12M users x 1 file = very long tail
– Too many connections
• 1M users + keepalive = 1M connections
– Living with modern content in web 2.0
• 1 file x (thumbnail + iPhone + Mac) = 3 file copies
Traditional Architecture
IO
IO
IO
IO
Traditional Architecture
IO
IO
IO
IO
Traditional Architecture
IO
IO
IO
IO
IO
IO
IO
Traditional Architecture
IO
IO
IO
IO
IO
IO
IO
Traditional Architecture
IO
IO
IO
IO
IO
IO
IO
“There are only two hard
things in Computer Science:
cache invalidation and
naming things”.
-- Tim Bray quoting Phil Karlton
Architecture goals
• Symmetric identical server nodes
– Simplified management and scaling
– Linear scaling out
• No functional / role servers
– No single point of failure
– No performance bottlenecks
• Multiple datacenters support
– DRP support
– Geo load distribution
Meet Prospero
• Distributed Web content storage system
• Full blown HTTP support
• Runs on low cost commodity hardware
• Adjustable file level replication controls redundancy
policy for every content type
• Provides dynamic image manipulation
How do we do it?
Designed to fail
• Fallback for every operation
– Geographical, machine, storage medium
• Write never fails
– All files will reach their destination
• Journaling
– Tracking all uploaded files
• Pending jobs
– Guaranteed file distribution
How do we achieve this
• Control the input
– define the only unified API
• Functional process isolation
–
–
–
–
every function deserves its own process by default
watchdogs
monitors
alerts
get 37D815B5.jpg
0.static
Go to 37 range servers
HTTP
2.static
20-3f
00-1f
HTTP
1.static
HTTP
Fallback if not found
4.static
6.static
60-7f
40-5f
HTTP
HTTP
3.static
HTTP
5.static
7.static
Fallback Example
Node Architecture
Output – Front
HTTP handler
Van Gogh
• lighttpd fork
• Dynamic image processor
• lighttpd fork
Cross Datacenter Distributer
• custom
Local Datacenter Distributer
• custom
Supervisor HTTP handler
Input – Supervisor
• Tornado web framework
application
Real Life
It’s all about performance
•
•
•
•
•
•
•
•
Non blocking IO, readiness notification (epoll)
Asynchronous file IO (AIO)
Zero copy (sendfile)
Memory maps
Inter-process binary protocols
UNIX socket
Minimize dynamic memory allocation
lighttpd memory footprint: 50MB
Lessons learnt
•
•
•
•
•
Be symmetric
Control the input
Design to failure
Performance matters again
Simple is hard but a must