報告_OCP2026_Oracle_GW級AI部署經驗_20261015

PDF 原檔：報告_OCP2026_Oracle_GW級AI部署經驗_20261015_original.pdf

講者公開簡報（OCP 官方 Recorded Content 頁提供下載）。文字以 PyMuPDF 逐頁抽取，### p<n> 為頁碼；圖表內容見原檔。

圖片清單（已驗證 2026-10-19）

原始內容

p1

Mark Filer Architect, Oracle Cloud Infrastructure Lessons learned from GW-scale AI deployments

p2

Oracle AI cluster growth 8x 131,072 GPUs 4x 65,536 GPUs 1x 16,384 GPUs 2x 32,768 GPUs 2020 2026 Enabled by: 1. RDMA at Zettascale with ultra-high throughput 2. Ultra-low latency 3. High network resilience 4. Advanced traffic load-balancing

p3

Network speed growth (NIC interfaces) 2017 2026 256x Network cluster performance increase 800 Gbps 1600 Gbps 400 Gbps 25 Gbps 50 Gbps 100 Gbps 200 Gbps

p4

Features • Standardized network interconnects, including LPO • Large-radix devices, non-blocking Clos • Multi-plane, rail-disjoint • 1.6T breakouts: 8x200G, 4x400G, 2x800G, 1x1600G • DR, FR, LR are all in scope Challenges • RDMA transport is lossless and sensitive to disruptions – network availability is critical • Network power envelope is non-negligible at Zettascale • Dust / contamination a primary source of link failure (attributed to 90% of link issues) Oracle RDMA multi-plane scale-out cluster https://blogs.oracle.com/cloud-infrastructure/first-principles-acceleron-multiplanar-networking

p5

Network Availability Power efficiency Network up-time, higher MFU, service credit avoidance Direct revenue impact. What are we solving for in next-gen networks? Less power ceded to network, more power for compute Direct revenue impact. Cost (CapEx) Relative spend of network to compute is increasing YoY Not a sustainable trajectory. At the end of the day, we’re solving for TCO…

p6

Addressing TCO at 800G, OCI has large deployment of LPO transceivers in AI scale-out networks Plots show 24-hr max pre-FEC BER across identical T0<>T1 links, 156k LPO and 202k FRO in data set(359k total) Observations • LPO marginally outperforms FRO in median pre-FEC BER • LPO greatly outperforms FRO in p99 pre-FEC BER Caveats • Short, med, and high loss ports require optimized tuning • Same switch ASIC for all (LPO + FRO) • Diverse optical DSPs for FRO 800G: Linear Pluggable Optics (LPO) Link type Valid links Median BER p99 BER LPO–LPO 156,591 1.137 × 10⁻¹¹ 5.165 × 10⁻¹⁰ FRO–FRO 202,042 1.432 × 10⁻¹¹ 2.888 × 10⁻⁸

p7

Top chart: corrected symbol rate as proxy for how “hard” the FEC is working looking at occupied FEC bins • Lower and farther left is better – fewer corrected symbols for the same amount of received traffic • FRO sits above and to right of LPO across most of the population, i.e., in general, FRO presents the decoder with more correctable symbol errors • At median, LPO is 3.1E-8 corrected symbols per codeword, while FRO is 1.6E-7 – roughly 5x higher correction activity Bottom chart: fraction of links that experienced at least one transition (i.e., flap) over a 1 month period • FRO links experienced 1.75x more transitions than LPO 800G: Linear Pluggable Optics (LPO) II Takeaway: Linear drive interfaces don’t introduce penalties, and in fact seem to provide better, more stable link performance for AI systems σ1 15(𝑏𝑖𝑛𝑖×𝑐𝑜𝑑𝑒𝑤𝑜𝑟𝑑𝑠𝑖) 𝑡𝑜𝑡𝑎𝑙 𝑐𝑜𝑑𝑒𝑤𝑜𝑟𝑑𝑠

p8

64x800G switch system-level power consumption Chart compares switches deployed with 100% FRO modules vs those with 50% LPO / 50% LRO: • Median full FRO power = 1528W • Median half LPO/FRO power = 1270W • Net savings = 258W 800G LPO modules save 5-7W compared to FRO. Expected savings due to modules alone: 32x ~6W = 192W Observed net savings (258W) vs savings due to modules (192W) indicate 66W additional due to system (fan) power reduction. 800G: Linear Pluggable Optics (LPO) III Takeaway: Confirmation that system-level power savings are compounded for LPO due reduced thermal burden of linear interfaces Median 1,270 W Median 1,528 W

p9

LPO cannot comfortably meet SI and interop performance requirements at 1.6T; LRO provides the best low power solution currently available: • 26dB channels at 16W power consumption (10 pj/bit) • Performs equally well over short, medium, long reach channels • Supports DR4+ (4dB), and FR4 links in the upper tiers • Interop has been demonstrated from different suppliers and multi-vendor 3nm DSPs • More aggressive power reduction expected as LRO DSP technology matures in 2nm. 1.6T: Linear Receive Optics (LRO) Takeaway: LRO is viable at 1.6T and markedly improves network TCO for AI scale-out applications configuration FEC bin p50 FEC bin p95 LRO 1 2 FRO 1 1

p10

Reliability / Availability • (Mis)handling of transceivers a primary failure mode – CPO removes humans from the deploy and ops process • System FIT expected lower than equivalent pluggable- based system1 Power efficiency: expected power savings of up to 50% over the equivalent fully-retimed pluggable-based system (200G/lane generation)2 Cost reduction: theoretical 30%+ cost efficiencies possible (assuming volume deployments)3 Beyond pluggables – CPO/NPO systems https://www.opencpxmsa.org NPO Source: Ciena CPO Source: Broadcom NPO offers some attractive features for end users and switch manufacturers, but must be proven to deliver reliability and power efficiency on par with soldered CPO 1. Evaluation of CPO performance and pluggable optics health for reliable AI infrastructure, S. Amiralizadeh (Meta), OCP 2025 2. https://developer.nvidia.com/blog/scaling-ai-factories-with-co-packaged-optics-for-better-power-efficiency 3. https://newsletter.semianalysis.com/p/co-packaged-optics-cpo-book-scaling

p11

Common failure modes • Dirt, dust, debris – airborne contaminants and mishandling, particularly in areas with high clay content in soil • Finger oils (sebum) – installer / technician mishandling • Grooves / etching – permanent damage requiring replacement of cable or transceiver Dust, dirt, and other contaminants Sampling of observed contaminants in actual deployments Some interesting stats from Oracle AI deployments • 90% of observed L1 link failures are contamination related • 97% of transceiver replacements are contamination

p12

EBO ferrule Expanded beam technology all but eliminates contamination as source of link down / flaps, and connector damage as a high-blast radius failure mode: • optical beam expanded for dust, debris, oil resilience • ferrule recessed to avoid direct contact with contaminants • some designs feature 90° turn to contain optical path within housing • no physical contact enables ~20x reduction in mating force, allowing large- format ganged insertion suitable for optical backplane applications Expanded beam optical technology Resilient connector design as an enabler of scale PRIZM® TMT EBO MPO-style connectors https://www.ebomsa.org/

p13

• It will take a multi-pronged approach to tackle all aspects of TCO for AI infrastructure – network availability, power efficiency, and capital expenditure • Linear drive interfaces are shown to improve on all of these metrics, paving the way forward for photonic integration (CPO/NPO) • Innovations like expanded beam technology will contribute further but like CPO/NPO, require multi-vendor, interoperable ecosystems to become a reality • Get involved! https://www.opencompute.org/ https://www.oiforum.com/ https://ebomsa.org https://opencpxmsa.org Call to Action

p14

Thank You! mark.filer@oracle.com