Prospero Media Storage Managing 100TB of small files… IGT – Event July 2011 Numbers • • • • • • • • 70TB used space 700 million files 200GB and 250,000 files uploaded every day 1200Mbps bandwidth throughput in peak 180TB of data is being served out monthly 3700 Hits per second in peak 40 storage node servers – 300TB raw space $0.13 per GB Motivation • Web 2.0 content serving paradigm shift – Too many files • 12M users x 1 file = very long tail – Too many connections • 1M users + keepalive = 1M connections – Living with modern content in web 2.0 • 1 file x (thumbnail + iPhone + Mac) = 3 file copies Traditional Architecture IO IO IO IO Traditional Architecture IO IO IO IO Traditional Architecture IO IO IO IO IO IO IO Traditional Architecture IO IO IO IO IO IO IO Traditional Architecture IO IO IO IO IO IO IO “There are only two hard things in Computer Science: cache invalidation and naming things”. -- Tim Bray quoting Phil Karlton Architecture goals • Symmetric identical server nodes – Simplified management and scaling – Linear scaling out • No functional / role servers – No single point of failure – No performance bottlenecks • Multiple datacenters support – DRP support – Geo load distribution Meet Prospero • Distributed Web content storage system • Full blown HTTP support • Runs on low cost commodity hardware • Adjustable file level replication controls redundancy policy for every content type • Provides dynamic image manipulation How do we do it? Designed to fail • Fallback for every operation – Geographical, machine, storage medium • Write never fails – All files will reach their destination • Journaling – Tracking all uploaded files • Pending jobs – Guaranteed file distribution How do we achieve this • Control the input – define the only unified API • Functional process isolation – – – – every function deserves its own process by default watchdogs monitors alerts get 37D815B5.jpg 0.static Go to 37 range servers HTTP 2.static 20-3f 00-1f HTTP 1.static HTTP Fallback if not found 4.static 6.static 60-7f 40-5f HTTP HTTP 3.static HTTP 5.static 7.static Fallback Example Node Architecture Output – Front HTTP handler Van Gogh • lighttpd fork • Dynamic image processor • lighttpd fork Cross Datacenter Distributer • custom Local Datacenter Distributer • custom Supervisor HTTP handler Input – Supervisor • Tornado web framework application Real Life It’s all about performance • • • • • • • • Non blocking IO, readiness notification (epoll) Asynchronous file IO (AIO) Zero copy (sendfile) Memory maps Inter-process binary protocols UNIX socket Minimize dynamic memory allocation lighttpd memory footprint: 50MB Lessons learnt • • • • • Be symmetric Control the input Design to failure Performance matters again Simple is hard but a must
© Copyright 2026 Paperzz