PCIe Over Cable Provides Greater Performance for Less Cost for High Performance Computing (HPC) Clusters from One Stop Systems (OSS) PCIe Over Cable PCIe provides greater performance 8 7 6 5 GBytes/s 4 3 2 1 0 EISA PCI PCI PCI-X AGP 32/33 64/66 64/133 8X Gb E- 10Gb net E-net Bus PCIe x1 PCIe x4 PCIe x8 PCIe x16 PCIe over Cable Comparison versus Ethernet 10 10 Gb Gb Ethernet Ethernet 9 3 to 80 times faster than 1Gb Ethernet PCIe Cost – Source OSS 9 Adapters: $100 to $700 PCIe PCIe over over Cable Cable 2.5Gb 2.5Gb to to 80Gb 80Gb Price PCIe Performance 9 Cables: $30 to $300 9 Switches: $600 to $1,200 PCIe cables 9 Heavy-duty well shielded cables 9 All cables are cross-over style 11 Gb Gb Ethernet Ethernet Performance PCIe best suited for small, local networks PCIe vs Infiniband 40Gb/s InfiniBand Bundle 9 36-port 40Gb/s InfiniBand switch 9 8 Single-Port 40Gb/s PCI Express 2.0 InfiniBand HCA cards 9 8 2 meters long, copper cables, 9 Total Price: $10K 80Gb/s PCIe Bundle 9 10-port 80Gb/s PCIe switch 9 80Gb host adapter and 2m cable 9 Eight 20Gb/s PCI Express 2.0 host adapters with 2m cables 9 Total Price: $6K PCI Express Basics: Two Architectures Tree – One CPU and multiple I/O boards CPU CPU CPU CPU I/O I/O Network – Multi CPUs, Multi I/O – Requires special H/W and S/W Switch I/O I/O I/O I/O PCI Express Basics Lanes – The Key to Performance CPU Tx Point-to-Point connections – no arbitration Each lane consists of two differential pairs 9 Separate Transmit and Receive pairs 9 2.5 or 5.0 Gb/s rate per pair 9 Components auto-detect max clock rate Rx x1 Lane I/O Multiple lanes are used to increase performance 9 9 9 9 x1 Î 5 Gb/s x1 (pronounced “by one”) x4 Î 20 Gb/s x8 Î 40 Gb/s x16 Î 80 Gb/s Tree Architecture – I/O Expansion Host system RAID array PCIe I/O expansion system PCIe switch CPCI/CPCIe IO expansion PCIe host cable adapters PCIe cost cable adapters PCIe x4 PCIe x8 PCIe x8 PCIe x16 PCIe cables Upstream Adapters PC, laptop and industrial form factors Downstream Adapters and Devices Creating downstream PCIe endpoints 9 PCIe board adapters 9 Backplane interface boards 9 Subsystems with PCIe cable inputs 9 Backplanes with PCIe cable inputs Direct Attached Expansion Kits Direct Attached Multi-port Switches Extends PCIe bus to multiple downstream sub-systems One upstream link to multiple downstream links Gen 1 and Gen 2 versions HPC requires Substantial infrastructure with: Long-life, redundant servers GPU accelerators for math co-processing High speed storage or Solid State Disk (SSD) appliances High-speed connectivity GPU Server 9 AMD-based MB 9 Eight GPU’s/SSD’s 9 Could also used as NAS SSD appliance 1U PCIe switch 9 One x16G2 uplink 9 Nine x4G2 downlinks Network connectivity Architecture Server to Switch 80Gb/s GPU Server 1U switch Server to Server at 20Gb/s 1U servers Two to eight servers 9 Server to server communication – – – 20Gb PCIe 10Gb Ethernet 20Gb Infiniband Multiple 1U or 2U GPU/SSD appliances 9 Two to eight GPU's/SSD’s per appliance 9 80Gb/s connectivity to server GPU/SSD appliance GPU/SSD appliance Server to appliance connectivity at 80Gb/s Latest server technology Longer life cycles from rugged servers reduce overall cost and downtime Reduced depth allows better fit in shallow racks Superior cooling and power Latest technology motherboards and processors provide wide range of processing options 9 Dual 5500-series “Nehalem” quad-core or six-core processors 9 Up to 96GB DRAM 2TB to 5TB disk drive capacity Server redundancy Network connectivity at 10-20Gb/s 1U PCIe switch 9 One 80Gb upstream interface 9 Nine 20Gb downstream interfaces Redundant servers with ExpressNet 9 Server to server communication 9 Windows or Linux OS 1U PCIe switch Multiple GPUs support many GPU Appliance users simultaneously in virtual networks 1U switch Server to Server at 20Gb/s 1U servers Appliance provides necessary cooling and power not found in servers for optimal operation and significant reduction in downtime GPU/SSD appliance Hot swappable appliances provide redundancy GPU/SSD appliance GPUs GPUs off-load high-end graphics and rendering from system processors GPUs provide rich media and 3D graphics to virtual desktops Typical GPUs 9 AMD FireStream™ 9270 – 9 • single precision: 1.2 TFLOPS • double precision: 240 GFLOPS AMD FireStream™ 9250 – 9 Processing power: Processing power: • single precision: 1.2 TFLOPS • double precision: 240 GFLOPS ATI Radeon™ HD 5870 – Processing power: • single precision: 2.72 TFLOPS • double precision: 544 GFLOPS Solid State Disk (SSD) appliance 1U appliance 9 One appliance per server 9 Up to four 640GB SSD boards 9 Server can access 2.5TB storage 2U appliance 9 One appliance per two servers 9 Up to eight 640GB SSD boards 9 Each server can access 2.5TB storage 1U or 2U GPU/SSD appliance 9 Up to 4 GPUs and 4 SSD boards per server 80Gb connectivity Each appliance can employ 4 to 8 Fusion-IO ioDrive Duo boards (for example) Combined GPU and SSD appliance Each 1U GPU/SSD appliance supports: 9 Up to 2 GPU’s per server 9 Up to 1.25TB SSD storage per server Each 2U GPU/SSD appliance 9 With 1U server supports: – Up to 2 GPU’s – Up to 1.25TB SSD storage 9 With 3U server supports: – Up to 4 GPU’s – Up to 2.5TB SSD storage 2U Integrated Server 10TFLOP server integrates 9 Motherboard with dual AMD six-core processors Up to four double-wide AMD 9270 or HD5870 GPUs Istanbul-based motherboard Up to four double wide or eight single wide GPUs 9 2.72TFLOPS each or 10TFLOPS total processing power Dual 1500 watt power supplies Four SATA/SAS hot swappable disk drives Superior cooling 9 12 chassis fans 9 4 power supply fans Four hot swap disk drives Dual 1500-watt power supplies PCIe connectivity Lower cost, lower latency, and less overhead than Network connectivity Server to Switch 80Gb/s Infiniband Greater throughput GPU Server/ 1U switch Server to Server at 20Gb/s and lower cost than 1U servers 10Gb Ethernet Server to server communication over 20Gb PCI Express GPU/SSD appliance 80Gb connectivity between GPU appliance and server GPU/SSD appliance Server to appliance connectivity at 80Gb/s The Future of HPC Clusters Network connectivity Latest technology server supporting up to eight GPU’s and/or SSD’s Long-life, latest technology, redundant 1U servers Server to Switch 80Gb/s 1U switch Server to Server at 20Gb/s 1U servers 20Gb/s PCI Express connectivity between servers GPU/SSD appliance GPU Server 1U or 2U GPU/SSD appliances with 80Gb/s connection to severs GPU/SSD appliance Server to appliance connectivity at 80Gb/s Direct Attached RAID Arrays 4-drive RAID PCIe x4 or 10Gb/s to RAID controller 12-drive RAID PCIe x8 or 20Gb/s to RAID controller 16-drive RAID PCIe x8 or 20Gb/s to RAID controller Possible Storage Configurations Host cable adapter PCIe expansion kit with RAID board Downstream cable adapter and backplane Possible Storage Configurations PCIe backplane with RAID board 3 PCIe x4 slots PCIex4 and x8 cable connectors GPU Computing/RAID Sub-System 80Gb PCIe 3U server 80Gb PCIe over cable 1U PCIe switch 1U GPU Accelerator (includes 2 GPU’s) 20Gb PCIe 20Gb PCIe RAID Array 1 20Gb PCIe 20Gb PCIe RAID Array 3 RAID Array 2 20Gb PCIe 20Gb PCIe RAID Array 5 RAID Array 4 20Gb PCIe 20Gb PCIe RAID Array 7 RAID Array 6 RAID Array 8 Global GPU/RAID System 10Gb Ethernet switches to the outside world 1Gb Ethernet connections between nodes Node 1 Node 3 1Gb Ethernet connections to redundant switches Node 2 Node 4 Summary PCI Express over cable operates from 10Gb/s to 80Gb/s Wide assortment of PCIe adapters, switches, and modules available PCI Express bus can be expanded from PC to IO or from PC to PC The expanding HPC market requires: 9 9 9 9 redundant servers, multiple GPU’s, high speed storage, and high-speed connectivity Servers with multiple high speed IO slots provide required bandwidth for GPU’s and high speed storage GPU’s and high speed storage can be attached to existing servers PCIe over cable provides the most economical high speed connectivity
© Copyright 2026 Paperzz