PDF 原檔：報告_OCP2026_OpenAI_Broadcom_MRC多路徑可靠連線_20261015_original.pdf
講者公開簡報（OCP 官方 Recorded Content 頁提供下載）。文字以 PyMuPDF 逐頁抽取，
### p<n>為頁碼；圖表內容見原檔。
圖片清單（已驗證 2026-10-19）
| 檔名 | 頁 | 分類 | 親眼所見內容 |
|---|---|---|---|
報告_OCP2026_OpenAI_Broadcom_MRC多路徑可靠連線_20261015_p2.png |
p2 | 真資料圖 | MRC 起源三階段（2024 初→10^6 節點目標→單一開放規格）：承襲 Ultra Ethernet Consortium 與既有 multipath／packet spray 設計；OpenAI 與 Microsoft 要求多路徑多平面；共同制定方為 AMD、Broadcom、Intel、Microsoft、NVIDIA、OpenAI |
報告_OCP2026_OpenAI_Broadcom_MRC多路徑可靠連線_20261015_p5.png |
p5 | 真資料圖 | 單一 MRC QP 經 Port A／Port B 分到 Plane A、Plane B，每個平面內有多條路徑（EV 03／11／21／29），逐封包選 EV；標示 multipath=路由多樣性、multiplane=埠與故障域多樣性 |
報告_OCP2026_OpenAI_Broadcom_MRC多路徑可靠連線_20261015_p12.png |
p12 | 真資料圖 | 正式環境 T1 交換器停止轉發事件：75K GPU 作業、約 25% QP 受影響、約 580K 封包遺失；時序為 T1 停轉→MRC 移除路徑→吞吐恢復→交換器重開 |
報告_OCP2026_OpenAI_Broadcom_MRC多路徑可靠連線_20261015_p13.png |
p13 | 真資料圖 | 實測曲線：吞吐約 12.5 Pb/s，T1 故障點（重開前約 12 分鐘）下探約 10.5 Pb/s 後約 1 分鐘內回到約 12 Pb/s；損失率在故障點尖峰（右軸約 0.0001 量級）後回落 |
原始內容
p1
Eric Spada Distinguished Engineer, Broadcom Guohan Lu Member of Technical Stuff, OpenAI Enabling AI Networking @ Scale with Multi-path Reliable Connections (MRC) AI Clusters
p2
MRC Origins and Design Objectives EARLY 2024 SHARED AMBITION OPEN COLLABORATION 10⁶ nodes DESIGN DIRECTION Proven foundations Diversity + resilience One open specification Ultra Ethernet Consortium work and earlier vendor designs for multipath and packet spraying OpenAI and Microsoft sought multipath and multiplane operation resilient to link and fabric failures AMD · Broadcom · Intel Microsoft · NVIDIA · OpenAI OCP MRC
p3
Application Semantics and Transport Mechanisms APPLICATION-VISIBLE CONTRACT Reliable delivery Connection-level reliability Ordered completions Disorder remains below the API Familiar model QP/CQ-style software abstractions PRESERVED MRC TRANSPORT MACHINERY Spray Per packet Bound Inflight state Recover Selectively Control Sender + paths Avoid Failed paths Mandatory primitives ensure correctness • Optional primitives add efficiency and resilience
p4
Limitations of RC at AI Scale PINNED One QP hashes to one ECMP path REPLAY One loss triggers go-back-N WAIT Failure recovery depends on timeouts and routing convergence RC QP Requestor RC QP Responder PLANE A SELECTED PLANE IMPAIRED LINK 5-TUPLE ECMP: ONE PATH PLANE B UNUSED BY THIS QP
p5
Multipath and Multiplane ONE MRC QP Per-packet EV selection PEER Same reliable connection PORT A PORT B PORT A PORT B PLANE A · MULTIPLE PATHS PLANE B · MULTIPLE PATHS EV 03 EV 11 EV 21 EV 29 Multipath: routing diversity • Multiplane: port and failure-domain diversity
p6
Source Routing With SRv6 ONE MRC ENTROPY VALUE ABSTRACTION The transport can use implicit hashing or explicit source routes CONVENTIONAL ECMP entropy Endpoint selects a hash input EXPLICIT Structured EV Hop-by-hop route encoded in packet fields EXPLICIT SRv6 uSID Path encoded as IPv6 micro-segments uS1 uS2 uS3 PATH CHOSEN BY THE NETWORK ROUTE IDENTIFIED BY EV ROUTE IDENTIFIED BY EV WHEN A ROUTE DEGRADES Identify Feedback maps to route Exclude Denylist unhealthy EV Reselect Use another known route NO NEED TO WAIT FOR ROUTING CONVERGENCE Source routing does not accelerate the routing protocol—it removes the endpoint’s dependency on it
p7
Bounded Reordering PACKETS MAY ARRIVE OUT OF ORDER REQUESTER RESPONDER PSN 105 PSN 103 PSN 106 RESPONDER-ADVERTISED MAXIMUM PSN RANGE MPR = 8 packets DELIVERED PREFIX ACK ≤ 102 WINDOW START UPPER EDGE 103 104 105 106 107 108 109 110 PSN 111 CANNOT SEND Bounded requester state Retransmission tracking ≤ MPR Bounded responder state Out-of-order tracking ∝ MPR Ordered completion preserved Disorder remains below the API Optional Dynamic MPR resizes the window as demand and available NIC resources change
p8
Selective Loss Detection and Recovery CONVENTIONAL RC Go-back-N replays the tail One missing packet invalidates later progress MRC Selective recovery preserves progress SACK identifies received packets and the actual gap 10 11 13 14 15 10 11 13 14 15 RETRANSMIT 12–15 SACK: 10 11 [GAP] 13 14 15 RETRANSMIT 12 DETECT EARLY · RECOVER PRECISELY · GUARANTEE PROGRESS SACK + NACK Explicit delivery state Mandatory reliability feedback Packet trim Header survives the drop Optional network acceleration Probe Elicit current SACK state Optional proactive query Timers + priority Fallback and fast repair Oldest gap retransmitted first Reordering is not loss. One loss should not erase later progress.
p9
Congestion Control Framework and NSCC STANDARDIZED BY MRC PER-CONNECTION POLICY IMPLEMENTATION BEHAVIOR SACK-carried feedback One feedback stream closes multiple loops ECN state Forward-path congestion marking Delivery accounting SACK clock and cumulative bytes received Network RTT RTT minus optional responder service time Responder backpressure Host and memory-subsystem congestion Reflected Entropy Value Correlates feedback with a specific path CE EV CC PROFILE Selects supported policy Configured per connection by the privileged controller TRANSPORT DEFINES THE SIGNALS Implementation defines how to react USED IN THIS IMPLEMENTATION NSCC Network Signaled Congestion Control Sender-based · window-driven · SACK-clocked Other algorithms May consume all or a subset of MRC signals CONTROL ACTIONS Byte window NSCC EV selection Load balancer SEND → MEASURE → ACK → ADJUST MRC standardizes interoperability signals—not one mandatory congestion-control algorithm
p10
Fast Path and Plane Failure Avoidance MRC QP REMAINS ACTIVE PEER Same connection PLANE A PLANE B EV DENIED INDEPENDENT RESPONSE PATH DATAPATH-TIMESCALE RESILIENCE EV state Exclude unhealthy routes Path-selection state EV probe Test one explicit path Path-scoped liveness Port Status Announce plane reachability Asynchronous update GID scoped Update many QPs at once Stateless endpoint operation Failure becomes a path-selection event—not a connection failure
p11
Application and Controller APIs APPLICATION API mrc.h Familiar verbs model Transport control and data-plane operations CONTROLLER API mrc_ctl.h Shared transport policy Privileged process with CAP_NET_ADMIN Reuse libibverbs Discovery and memory registration Familiar resources mrc_qp and mrc_cq lifecycle Discover capabilities Device and per-port feature queries Configure and transmit Write and Write-with-Immediate Query capabilities Mandatory device and port discovery Create EV profiles Shared path universe and selection policy Create CC profiles Shared congestion-control policy Optional path controls EV-state events and EV probes query_device · query_port · modify_qp Policy and telemetry shared across many QPs DATA PLANE Reliable connection OPERATE POLICY BINDS TO EV PROFILE CC PROFILE Connection setup remains out of band Best-effort Ethernet RoCEv2 / UDP / IP encapsulation Familiar application semantics • centralized path and congestion policy
p12
Cisco Confidential What happened when a T1 stopped forwarding 75K GPUs in the production job ~25% of QPs affected ~580K packets dropped T1 stops forwarding MRC removes path throughput recovers switch reboot Resilient AI Supercomputer Networking using MRC and SRv6
p13
Real data captured during a training run showing MRC reacting to the complete loss of a T1 switch. The training job experienced a temporary slowdown, but recovered quickly. A T1 switch failure and reboot has minimal impact on training throughput −14 −12 −10 −8 −6 −4 −2 0 2 4 Throughput (Pb/s) Loss rate (percent) Time (minutes from reboot) THROUGHPUT SWITCHED FAILED SWITCH REBOOTED SWITCH UP LOSS RATE 14 12 10 8 6 4 2 0 0.001 0.0008 0.0006 0.0004 0.0002 0
p14
• Where to find additional information (URL links) • MRC 1.0 Spec http://www.opencompute.org/documents/ocp-mrc-1-0-pdf • MRC API github: https://github.com/opencomputeproject/OCP-Multipath-Reliable- Connection [Call to Action]
p15
（本頁無可抽取文字，內容在圖中）