PDF 原檔：報告_OCP2026_Meta_AMD_MetaRoCE_20261015_original.pdf
講者公開簡報（OCP 官方 Recorded Content 頁提供下載）。文字以 PyMuPDF 逐頁抽取，
### p<n>為頁碼；圖表內容見原檔。
圖片清單（已驗證 2026-10-19）
| 檔名 | 頁 | 分類 | 親眼所見內容 |
|---|---|---|---|
報告_OCP2026_Meta_AMD_MetaRoCE_20261015_p8.png |
p8 | 真資料圖 | AMD 可程式化 AI NIC 方塊圖：PCIe、P4DMA（Tx/Rx PDMA、SQ/RQ）、Packet Buffer、MAC（800Gbps uplink）、P4NET（Parser／Deparser）、可程式化多核封包處理器與記憶體引擎 |
報告_OCP2026_Meta_AMD_MetaRoCE_20261015_p9.png |
p9 | 真資料圖 | 同時 scale-out 與 scale-across：Site 1（本地）Node-0／1 與 Site 2（遠距）Node-2，RTT 1.5ms；表列 scale-out RTT 5–15µs、Max Cwnd 約 75、Max Path 約 12；scale-across RTT 常達毫秒（50–100 倍）、Max Cwnd 3500–7000、Max Path 約 30–50（4K MTU） |
報告_OCP2026_Meta_AMD_MetaRoCE_20261015_p11.png |
p11 | 真資料圖 | MetaRoCE 在 multiplane FPF 網路：兩台 AI Rack Platform（PCIe Gen6、AMD AI-NIC）各以 8×100G 連 Plane 0–7；下方左圖 8×100G multiplane 吞吐（單向約 800Gbps、雙向約 1,520Gbps，約 16KB 後飽和），右圖 plane failure 時吞吐由約 780 降到約 680Gbps，約 65 秒後回升 |
報告_OCP2026_Meta_AMD_MetaRoCE_20261015_p12.png |
p12 | 真資料圖 | MetaRoCE vs RoCEv2 在 64 GPU 的集合通訊完成時間差異（相對 baseline）：All Reduce 多數訊息大小約減 5% 至 27%；All To All 小訊息約減 15–28%，少數大訊息（如 1MiB、32MiB、4–16GiB）反而增加約 5–20% |
報告_OCP2026_Meta_AMD_MetaRoCE_20261015_p13.png |
p13 | 真資料圖 | MetaRoCE 開源生態循環圖：Specification、Reference Software、Compliance Test Suite 三者開源，與 MetaRoCE on Vendor Hardware 形成持續精進迴圈 |
原始內容
p1
Arvind Srinivasan, Meta Platforms Balakrishnan Raman, AMD Sandeep Nagaraja, Meta Platforms MetaRoCE RDMA Transport NETWORKING
p2
AI Network Transformation The network is in the critical path for AI Clusters — no longer auxiliary but an integral part of infrastructure SCALING CHALLENGES ARE REAL Training Bottlenecks All-reduce and all-to-all synchronization — performance gated by the slowest node. Inference Latency Latency between distributed model shards directly impacts response times for hundreds of millions of users. Massive Cluster Size Hundreds of thousands of accelerators today, across multiple data centers and regions — designed for a million. AI NETWORK SCALE OVER TIME NUMBER OF ACCELERATORS
p3
Optimized RDMA Transport for Meta’s Network Endpoints & AI Accelerators Meta-RoCE Transport PERFORMANCE @SCALE
90% network efficiency in non-blocking manner Per packet load balancing & spray RESILIENCE @SCALE Reliably scaling across multiple network planes with transparent failover Per-path reliability coupled with congestion control OPERATIONS @SCALE Operates on commodity Ethernet Infrastructure Frictionless adoption with full verb compatibility
p4
Multipath RDMA transport for AI Clusters ● Per-path reliability and congestion control ● Receiver rate hints (RCN) for fast incast convergence ● Tolerates packet loss and reordering ● Single QP carries multiple streams over multiple network paths ● Runs on commodity Ethernet: ECN + ECMP, no switch modifications Meta-RoCE Transport
p5
Fundamentals Key Architectural Decisions Use Commodity Ethernet Switch ● ECN and ECMP only — no trimming, adaptive routing, or in-network telemetry ● Trimming adds switch and NIC complexity; SACK detects loss without either PFC, go-back-N, and reorder buffers do not scale ● PFC headroom grows with distance; deadlock risk in large fabrics ● Go-back-N wastes bandwidth; reorder buffers cost SRAM that grows with BDP ● Instead: Order completions, Not delivery — OOO placement, no reorder buffer
p6
MetaRoCE Paths: Building Block for Multipath Activated on Failures QWND = ΣPWND[p] QP QP PWND[1] PATH ID 1 PWND[2] PATH ID 2 PWND[3] PATH ID 3 PWND[1] PATH ID 1 PWND[2] PATH ID 2 PWND[3] PATH ID 3 RTT RTT RTT RECEIVER Rate Feedback for QP Through ACK (in any path) PWND[3] PATH ID 3 PWND[3] PATH ID 3 RTT Reliability & Path Window Receiver Memory M0 M1 M2 M0 M1 M2 Application Messages
p7
Congestion Control & Reliability QWND Update Additive Increase Multiplicative Decrease Multiplicative Decrease ACK (in any path) Congestion Experienced N Packets Drop Signals QWND = ΣPWND[p] QP QP PWND[1] PATH ID 1 PWND[2] PATH ID 2 PWND[3] PATH ID 3 PWND[1] PATH ID 1 PWND[2] PATH ID 2 PWND[3] PATH ID 3 RTT RTT RECEIVER Rate Feedback for QP Through ACK (in any path) PWND[3] PATH ID 3 PWND[3] PATH ID 3 ECMP Fabric
p8
AMD Programmable AI NIC
p9
Simultaneous Scale-out and Scale-across NODE-2 NODE-0 FE CP FE AI NIC AI NIC AI NIC AI NIC NODE-1 GPU GPU GPU GPU GPU GPU GPU GPU SITE 1 (Local) SITE 2 (Long Distance) Scale-Out Scale Across RTT: 5-15 micro-seconds RTT: Often reaches milli-seconds i.e. 50-100x Max Cwnd: ~75; Max Path: ~12 (4k MTU packets); Line rate performance Max Cwnd: 3500 to 7000 Max Path: ~30 - 50 (4k MTU packets) Line rate performance AI NIC AI NIC AI NIC AI NIC FE CP FE AI NIC AI NIC AI NIC AI NIC GPU GPU GPU GPU GPU GPU GPU GPU AI NIC AI NIC AI NIC AI NIC FE CP FE AI NIC AI NIC AI NIC AI NIC GPU GPU GPU GPU GPU GPU GPU GPU AI NIC AI NIC AI NIC AI NIC RTT: 1.5ms
p10
MetaRoCE Congestion Control: Incast Handling NODE-2 NODE-0 FE CP FE AI NIC AI NIC AI NIC AI NIC NODE-1 GPU GPU GPU GPU GPU GPU GPU GPU Multiple Senders Receiver AI NIC AI NIC AI NIC AI NIC FE CP FE AI NIC AI NIC AI NIC AI NIC GPU GPU GPU GPU GPU GPU GPU GPU AI NIC AI NIC AI NIC AI NIC FE CP FE AI NIC AI NIC AI NIC AI NIC GPU GPU GPU GPU GPU GPU GPU GPU AI NIC AI NIC AI NIC AI NIC - Congestion window rate-hints(in Gbps) : 200 Gbps per sender
p11
MetaRoCE on a Multiplane FPF Network Multiplane Switch Fabric Disjoint Planes 8x100G Planes Plane 0 Plane 1 Plane 2 Plane 3 Plane 4 Plane 5 Plane 6 Plane 7 Disjoint Planes 8x100G Planes Plane 0 Plane 1 Plane 2 Plane 3 Plane 4 Plane 5 Plane 6 Plane 7 AI Rack Platform PCIe Gen6 AMD AI-NIC AMD AI-NIC 8x 100G 8x 100G AI Rack Platform PCIe Gen6
p12
MetaRoCE vs RoCEv2 Collective Completion Times
p13
Enabling OCP community on MetaRoCE development
p14
Call to Action Make large-scale AI faster, cheaper, and truly open for everyone. 1. Adopt & Contribute Join us in making MetaRoCE an open, multi-vendor AI fabric. ● Adopt the open spec using Ref Implementation. ● Implement & test against SmartNICs, DPUs, switches, and ASIC designs. ● Publish your results. 2. Validate & Standardize Help achieve broad hands-on validation, interoperable hardware, and shared operational experience. Together, we can turn a purpose-built transport into an industry standard.
p15
Thank You!