PDF 原檔：報告_OCP2026_Microsoft_MRC_AI訓練RDMA傳輸_20261015_original.pdf
講者公開簡報（OCP 官方 Recorded Content 頁提供下載）。文字以 PyMuPDF 逐頁抽取，
### p<n>為頁碼；圖表內容見原檔。
圖片清單（已驗證 2026-10-19）
| 檔名 | 頁 | 分類 | 親眼所見內容 |
|---|---|---|---|
報告_OCP2026_Microsoft_MRC_AI訓練RDMA傳輸_20261015_p4.png |
p4 | 真資料圖 | 靜態雜湊（ECMP）對比自適應多路徑示意：左側 4 條流中 3 條撞同一路徑、其餘容量閒置；右側 4 條流分散到全部路徑 |
報告_OCP2026_Microsoft_MRC_AI訓練RDMA傳輸_20261015_p6.png |
p6 | 真資料圖 | Fungible Fabric Control Protocol：Tx Host→DPU→ToR→Spine→ToR→DPU→Rx Host 架構與 fabric 延遲分布，圖下註明 99p 10.5us、99.9p 13us |
報告_OCP2026_Microsoft_MRC_AI訓練RDMA傳輸_20261015_p10.png |
p10 | 真資料圖 | 正式環境 Cluster A 規格條列（GB200、ConnectX-8 800Gb/s、兩層 Clos、4×200G 平面、Spectrum-4／Tomahawk-5 混用）；右上為 250 分鐘內 T0–T1 link flap 次數（約 1–13 次）；右下 Cluster B 雙向頻寬約 1,550 Gb/s，20 條連結失效與恢復期間幾乎不變 |
報告_OCP2026_Microsoft_MRC_AI訓練RDMA傳輸_20261015_p15.png |
p15 | 真資料圖 | MRC 互通性三級需求表：Level 1 僅標頭欄位一致；Level 2 加 SMaRTT、ReSS、禁用 Go-Back-N、per-packet／連線 RTO；Level 3 再加窗口增減速率與故障反應時間一致 |
原始內容
p1
MRC: A Purpose-Built RDMA Transport for AI Training Networks NETWORKING Abdul Kabbani Partner, Microsoft Wael Noureddine VP, Microsoft
p2
• Foundations o Multipathing o Congestion control o Loss recovery • MRC in action at scale • MRC moving forward Overview
p3
Background: Fundamentals and Key pre-MRC Transport Efforts
p4
Static Hashing Causes Network Imbalance and Hotspots The transport must react when equal-cost paths are not equally loaded. ECMP: one flow, one fixed path Flow 1 Flow 2 Flow 3 Flow 4 Adaptive multipath: Continuously rebalance good paths Balanced utilization and fast reroute Idle capacity Flow 1 Flow 2 Flow 3 Flow 4 3 collisions, congestion Idle capacity
p5
Why Multipathing Is Foundational to AI Transport AI fabrics expose abundant paths, but transports must use them intelligently. 1. Path diversity Fat-tree networks provide many equal-cost paths. A single fixed path per connection leaves capacity stranded. 2. Dynamic imbalance Collisions, incast, asymmetry, and failures shift the best path over time. 3. Transport response Multipath makes path choice a responsive transport function, not a static routing outcome.
p6
• First production packet spray DC transport • Expands high-end router fabric design to DC scale • End-to-end flow and congestion control • Reliable underlay transport – designed to carry RDMA, TCP, UDP… • Two-tier full cross-section bandwidth network topology Fungible’s Fabric Control Protocol All-to-all traffic pattern, fabric latencies are short-tailed, with a 99p value of 10.5us and 99.9p value of only 13us.
p7
Must Co-design Load Balancing and Congestion Control Path placement determines the signal seen by congestion control, and vice versa. Load balancing Question: Where should the next packet go? Goal: spread traffic over available capacity Signal: ECN, delay, and trimming Congestion control Question: How much data should be in flight? Goal: control queues while sustaining throughput Signal: ECN, delay, and trimming Path quality Sending rate MRC uses focused mechanisms that cooperate rather than treating these as independent problems.
p8
SMaRTT: Fast, Multi-Signal Congestion Control Window-based control combines ECN, delay, and optional trimming. Fast increase Underloaded Proportional Increase Uncongested Multiplicative Decrease Congested Fast Convergence Overload Inputs ECN: leading indicator RTT: multi-bit congestion level Trimming: rapid overload response Design intent: responsive during incast and ramp-up, gradual in steady state, simple enough for hardware.
p9
Reactive Subflow Spraying (ReSS) A small set of subflows provides path agility while preserving useful ordering. Sender Maintains a small active subflow set Fabric Multiple paths Receiver Returns ECN / RTT feedback subflows feedback identifies congestion; sender reroutes future packets • Randomly reroute the most congested subflow • Congestion control at the flow level • Few active subflows can still converge across many available paths.
p10
• Cluster A is a large production AI-training supercomputer o Compute: NVIDIA GB200 GPUs, used to run 50K- and 75K- GPU training jobs o NICs: NVIDIA ConnectX-8, each providing 800 Gb/s. o Fabric: Two-tier Clos network with T0 and T1 switches. o Planes: Four independent 200-Gb/s planes. Each NIC’s 800-Gb/s connection is split into 4 × 200 Gb/s links. o Switches: A mixture of NVIDIA Spectrum-4 and Broadcom Tomahawk-5. o During production training, frequent T0–T1 link flaps occur as shown in the figure on the right, yet MRC transparently reroutes traffic and keeps their impact on the job negligible. • Cluster B is an experiment demonstrating bi- directional throughput given an accelerated 20-link-failure scenario MRC is Deployed on Production AI-Training Supercomputers
p11
MRC: Moving Forward
p12
Towards Heterogenous MRC Deployments • Early MRC deployments assumed a single NIC vendor across the fleet • Broad adoption requires interoperability across multiple NIC implementations • Mixed-vendor fabrics are becoming the norm for AI inference networks
p13
Example MRC Specification Gaps Reserved bits, unclear ownership Some implementations repurpose “reserved” header bits for vendor-specific signaling (e.g., plane health). The spec doesn’t say if “reserved” means reserved-for-vendor or reserved-for-future-use, so wire-format compliance becomes ambiguous Reliability algorithm unspecified The spec doesn’t define reliability/retransmission behavior, which directly affects RTO handling — reducing interoperability effectiveness even when wire format matches RTT measurement left open MRC spec doesn’t state whether RTT should be derived from data packets, SACKs, or dedicated probe packets — so implementations diverge on how (and how often) they sense congestion
p14
Defining Interoperability Requirements What does “interoperable” actually mean? Full interoperability requires Level 3. LEVEL 1 Wire-format compatibility A common packet format — headers, opcodes, and field semantics — so vendors can at least parse each other’s packets LEVEL 2 Algorithm compatibility Comparable load balancing, congestion control, and reliability logic LEVEL 3 Behavior compatibility Matching signal frequency, reaction latency, and timing dynamics, so mixed-vendor endpoints perform well in the shared network
p15
MRC Interoperability Requirements Requirement LEVEL 1 LEVEL 2 LEVEL 3 Identical header field sizes and offsets SMaRTT congestion control (CC) ReSS load balancing (LB) No Go-Back-N retransmission Per-packet / per-connection retransmission timeout (RTO) CC grows the congestion window at the same rate when uncongested CC shrinks the congestion window equally at equal congestion Same LB reaction time to link failures Each level builds on the one below it; full interoperability requires Level 3.
p16
Building Consensus & Planning the MRC Plugfest Build Consensus • Invite vendors and operators to share implementation experience and known interop pain points • Present the identified gaps and the Level 1–3 requirements as a shared starting point • Align on which gaps are must-fix vs. nice-to-have • Agree on how decisions get ratified within the OCP networking community Plan the Plugfest • Stand up a shared interoperability lab / testbed for cross-vendor validation • Define a concrete test suite — e.g., congestion- control test cases and failure/degradation scenarios • Agree on a process and timeline for an initial cross- vendor plugfest
p17
Interoperability • Hone the specs so basic interoperability is guaranteed Evolution • Transport management should be malleable as workloads evolve; best to furnish unified programmable congestion management solutions. Generality • Expand specs to encompass semantics beyond only writes Call for Action
p18
Open Discussion