報告_OCP2026_Marvell_XPU級可組合記憶體Fabric_20261015

PDF 原檔：報告_OCP2026_Marvell_XPU級可組合記憶體Fabric_20261015_original.pdf

講者公開簡報（OCP 官方 Recorded Content 頁提供下載）。文字以 PyMuPDF 逐頁抽取，### p<n> 為頁碼；圖表內容見原檔。

圖片清單（已驗證 2026-10-19）

原始內容

p1

Gaurav Agarwal Senior Distinguished Engineer, Marvell Satananda Burla Distinguished Engineer, Marvell XPU Scale Composable Smart Memory Fabric SERVER: COMPOSABLE MEMORY SYSTEMS (CMS) © 2026 Marvell. All rights reserved.

p2

Except for statements of historical fact, this presentation contains forward-looking statements (within the meaning of the federal securities laws) including statements related to future revenue, future earnings, and the success of our product releases that involve risks and uncertainties. Words such as “anticipates,” “expects,” “intends,” “plans,” “projects,” “believes,” “seeks,” “estimates,” “can,” “may,” “will,” “would” and similar expressions identify such forward-looking statements. These statements are not guarantees of results and should not be considered as an indication of future activity or future performance. Actual events or results may differ materially from those described in this presentation due to a number of risks and uncertainties. Forward-looking statements are only predictions and are subject to risks, uncertainties and assumptions that are difficult to predict, including those described in the “Risk Factors” section of our Annual Reports on Form 10-K, Quarterly Reports on Form 10-Q and other documents filed by us from time to time with the SEC. Forward-looking statements speak only as of the date they are made. You are cautioned not to put undue reliance on forward- looking statements, and no person assumes any obligation to update or revise any such forward-looking statements, whether as a result of new information, future events or otherwise. Forward-looking statements © 2026 Marvell. All rights reserved.

p3

Composable memory solutions © 2026 Marvell. All rights reserved. xPU Memory Accelerator xPU Memory Expander Direct Attach Scale performance and capacity Memory accelerator Structera X Structera A Structera P 56 Core xPU xPU Memory Pooling Disaggregated Server Share memory across xPU Structera S Memory expanders Memory accelerators xPUs xPU 0 xPU N … CXL Switch CXL pooling … … Rack scale composability Disaggragted Rack Photonic Fabric Memory Appliance ….. Photonic Fabric Memory Appliance  Multi-Rack/Pod-Scale optically connected shared memory  Unified memory for RAG/Agentic AI in CPU/XPU clusters TM TM

p4

Photonic Fabric Memory Appliance 16 servers PF-NIC PCIe SW CPU CPU XPU XPU XPU XPU x16 PF Appliance Back-end network Ultra low-latency access to PF Appliance (<350ns) 1TB HBM3 + up to 32TB DDR5 Photonic Fabric Memory Appliance Typical server-to-server latency = ~ us 2 Tbps* 2 Tbps 2 Tbps 2 Tbps 16 Compute servers Rack mountable, pod-scale memory appliance, supports CXL 3.1 (incl. CXL DCD) Memory capacity: 1TB HBM3e + 128x standard DDR5 DIMMs (16/32TB) Connects 16 compute servers or 16 XPUs or combination with one PFMA Unified shared address space with DCD © 2026 Marvell. All rights reserved.

p5

Structera S : CXL switch for rack scale pooling Structera S CXL switches PCIe 5.0 16x16 | 32x8 CXL 2.0 1.6nm 2TB/s Switching capacity Intel and AMD Server CPUs Up to 100TB CXL memory pool Sub-100ns Port-to-port latency 135W Memory pooling and sharing 2x 80-bit DDR5 2DPC 2x 80-bit DDR5 2DPC CPU Structera S CPU … Memory pooling Peer-to-peer I/O Ultra IO Transformer © 2026 Marvell. All rights reserved.

p6

Memory tiers for KV$ offloading (updated) G1 – GPU HBM ~nanoseconds | 100s of GB G2 – system DRAM ~10-100 nanoseconds staging | spillover KV | 1-2 TB G3 – local SSD/rack-local ~Microseconds | 10s of TB Lower latency Higher inference performance Increasing latency Lowest efficiency G4 – shared object/file ~Microseconds | persistent data | 100s of TB G3.5 High-performance flash G2.5 – CXL fabric attached memory (Structera or PFMA)| 10s of TB @ 100s of ns G1.5 – PFMA w/ PF-Chiplet (package level integration) | 10s of TB @100s of ns © 2026 Marvell. All rights reserved.

p7

XPU scale smart memory fabric Compute cluster Access AI data objects in disaggregated memory via API Structera A Structera A Structera A Structera A XPUs High-performance data-path Peer-to-peer PCIe UlOT: PCIe/CXL hybrid switching Disaggregated memory w/ compute Native AI objects aware Provides data as a service Structera S © 2026 Marvell. All rights reserved.

p8

Offloaded Data Services: KV cache service © 2026 Marvell. All rights reserved.

p9

KV cache service offload to smart memory fabric © 2026 Marvell. All rights reserved. Complete KV cache service offload to the fabric Memory efficient shared model Store once read multiple Cache tier scales independent of compute tier Cache availability is independent of compute No synchronization with compute overhead Simple load-based routing – avoids complexity Equidistant to compute – predictable performance KV$ optimizations naturally fits Near Memory Compute LM Cache I/F LM Cache I/F LM Cache I/F LM Cache I/F App server Router User KV$ Structera A KV$ Structera A KV$ Structera A . . . . . . PCIe/CXL KV$ Service API Logical blocks Prefetching Eviction Promotion / demotion Custom policies Custom backends Lookups Hash computation Metadata mgmt. State management Serialization Deserialization Dataplane API Storage providers Transports / Data movers Data management Storage management Scoring Quantization Compaction: Pruning Chunking Blending KV Optimizations Highlights Agentic AI Inferencing System

p10

KV$ service offload performance (early results) Raw transport bandwidth (GB/s) read & write · higher is better 31.8 46.8 50.0 31.1 46.9 48.0 0 10 20 30 40 50 Local SSD RDMA Structera A Solution Bandwidth GB/s Read Write End-to-end inference throughput lmbench long-doc-qa · Peak users 16 · Llama-3.1-8B · H200 TTFT: Time To First Token (inference latency) 5× lower TTFT → 5.9 to 1.2 s 4× higher throughput → 31k tok/s to 139k tok/s 17× lower read latency vs Local SSD 52us → 3us input tok/s · higher is better · TTFT · lower is better 31,394 44,318 80,813 139,497 5,933 4,243 2,162 1,199 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 HBM (Recompute) SSD DRAM (CPU Load) Structera A Solution Mean TTFT (ms) Number of tokens/s Input tok/s Mean TTFT (ms) © 2026 Marvell. All rights reserved. Source: Marvell

p11

L1 (uncompressed, fast) L2 (compressed, secondary) Custom compute logic L1 to L2 eviction path L2 to L1 cache hit path Host Node (CPU + GPU) Structera A vLLM LLM inference engine LMCache KV cache connector : lookup / store GPU Holds KV cache in HBM DRAM /dev/shm : compressed KV data (per-chunk serde) ARM Compute (Neoverse V2) Near-memory compute : runs serde + eviction logic Adaptive Compression Selector None 1.0x FP8 2.0x CacheGen 3-4x TurboQ 3x-.4x User Select Selects per-chunk based on: reuse frequency, L2 capacity, user config or explicit UserSelection override L1 Eviction Trigger LRU eviction when L1 exceeds watermark LMCache MP Server KV cache service (store controller) G2.5 Storage DRAM : uncompressed KV cache (devdax / CXL-mapped) Control Plane ZMQ / TCP Data Plane PCIe / CXL DMA Algorithmic optimizations: KV$ compression © 2026 Marvell. All rights reserved.

p12

KV$ compression offload performance Compression Method Mean TTFT ↓ Input tok/s ↑ L2 Size Occupied ↓ Compression Ratio ↑ KV Cache Hit Rate ↑ ARM Avg Cores ↓ Accuracy Loss ↓ None (raw BF16) 4864 ms 16,832 8000 MB 1.0x 35.1% 0.5 0 FP8 (float8_e4m3fn) 3374 ms 22,860 3328 MB 2.0x 33.4% 2 1.52 TurboQuant k8v4 2914 ms 22,845 1884 MB 2.61x 46.5% 7.4 3.9 CacheGen 2834 ms 25,594 278 MB 5.6x 34.8% 0.6 13.4 G2.5 Storage: DDR size 6GB G3 Storage: Secondary storage for storing compressed data 12GB Benchmark Environment: Model Used: meta-llama/Llama-3.1-8B-Instruct Dtype: torch.bfloat16 Workload: lmbench long-doc-qa 4 documents x 64K-token context x queried 10 rounds (tile-shuffled) 4 x in-flight concurrently Hardware: 1xH200(27% mem_util), 1xStructeraA (32GB + 32GB) Software: vLLM + LMCache, LMCache MP Server Note: ↓: Lower the better ↑: Higher the better CacheGen: Implementation incomplete Accuracy Loss: Cosine Similarity Angle (0 equal, 90 all not equal) ARM Avg Cores: Used ARM Neoverse V2 out of 16 TTFT: Time To First Token (Inference Latency) tok/s : Input Tokens processed per second © 2026 Marvell. All rights reserved. Source: Marvell

p13

Data Transfer Across Fabric © 2026 Marvell. All rights reserved.

p14

Structera ODM for GPUDirect transfers GPU HBM (H200 NVL) ⚡ PCIe P2P DMA Expander DMA controller initiates transfers directly against GPU HBM — GPU is passive Full Host Bypass Data never touches host DRAM or traverses host IOMMU — pure switch- domain transfer Framework Compatible Integrates via a custom NIXL backend plugin — frameworks using NIXL require no architectural changes Key Insight: GPU-facing traffic routes over PCIe P2P DMA (no CXL needed on GPU side). © 2026 Marvell. All rights reserved.

p15

KV cache transfer: Bidirectional data path Write: GPU DMA DAX plugin · Read: Structera DMA ODM plugin Host Node (CPU + GPU) vLLM LLM inference engine LMCache KV Cache Management, KV Data-transfer layer GPU Holds KV cache in HBM Contiguous KV blocks · 2MB/block Data Plane Structera X 2404 DRAM Memory Up to 12 DIMMs · 1.5 TB Inline Compression LZ4 ~1.6–2× at line rate DMA KV in/out transport Block 0 Block 1 Block 2 … Block N Write (push) GPU DMA memcpy (DAX plugin) Read (push) Structera DMA (ODM Plugin) © 2026 Marvell. All rights reserved.

p16

Nixlbench bandwidth: RDMA vs Structera 50.6 46.3 0.0 GB/s 15.0 GB/s 30.0 GB/s 45.0 GB/s 60.0 GB/s 4 KB 8 KB 16 KB 32 KB 64 KB 128 KB 256 KB 512 KB 1 MB 2 MB 4 MB 8 MB 16 MB 32 MB 64 MB Read Bandwidth GB/s · GPU reads · by block size Structera→GPU Remote→GPU (RDMA) 55.5 46.3 0.0 GB/s 15.0 GB/s 30.0 GB/s 45.0 GB/s 60.0 GB/s 4 KB 8 KB 16 KB 32 KB 64 KB 128 KB 256 KB 512 KB 1 MB 2 MB 4 MB 8 MB 16 MB 32 MB 64 MB Write Bandwidth GB/s · GPU writes · by block size GPU→Structera GPU→Remote (RDMA) © 2026 Marvell. All rights reserved. Source: Marvell

p17

Nixlbench latency: RDMA vs Structera 1,325 23,199 1 µs 10 µs 100 µs 1,000 µs 10,000 µs 100,000 µs 4 KB 8 KB 16 KB 32 KB 64 KB 128 KB 256 KB 512 KB 1 MB 2 MB 4 MB 8 MB 16 MB 32 MB 64 MB Read Latency µs · GPU reads · by block size Structera→GPU Remote→GPU (RDMA) 1,209 23,197 1 µs 10 µs 100 µs 1,000 µs 10,000 µs 100,000 µs 4 KB 8 KB 16 KB 32 KB 64 KB 128 KB 256 KB 512 KB 1 MB 2 MB 4 MB 8 MB 16 MB 32 MB 64 MB Write Latency µs · GPU writes · by block size GPU→Structera GPU→Remote (RDMA) https://www.hotstorage.org/2026/accepted.html# © 2026 Marvell. All rights reserved. Source: Marvell

p18

• CMS and CMS/CxP working with FTI DCC, Startups, and SNIA • Exploring open-formats based stable API for computational devices that aligns to AI frameworks and middleware • Explore a unified data-service API providing a consistent interface for XPU and AI framework for AI-native data services • Academia • Advance Nead Data Processing architectures, open prototypes, and workload-driven benchmarks • Hyperscalers • Co-design and validate memory-fabric solutions against real-world AI serving workloads • Resources https://www.opencompute.org/projects/composable-memory-system https://www.opencompute.org/projects/data-centric-computing Call to Action © 2026 Marvell. All rights reserved.

p19

Thank You! © 2026 Marvell. All rights reserved.

p20

Early benchmarking: workload 24 GB --kv-cache-volume Total KV data generated — about 3 documents of 64K tokens each (at 8,192 tokens/GB) 64,000 tokens --ldqa-document-length Each synthetic document is ~64K tokens long 20 rounds --ldqa-query-per-document Each document is queried 20 times (repeated Q&A over the same long context) 4 --ldqa-num-inflight-requests 4 concurrent users hitting the engine simultaneously tile --ldqa-shuffle-policy Interleaves requests across documents in a tile pattern (not pure round-robin, not fully random) 8,192 --tokens-per-gb-kvcache Conversion factor used to size the working set Benchmark: lmcache bench Workload: A shared long-document Q&A scenario (a small set of large documents queried repeatedly by multiple concurrent users) © 2026 Marvell. All rights reserved. Source: Marvell

p21

Early benchmarking: configuration TEST SYSTEM CONFIGURATION HOST GPU: 1xNVIDIA H200 NVL CXL mapping: /dev/dax0.0, 32 GiB devdax window CUDA: CUDA 13 driver Boot param: iommu=pt ARM NMC: 16x aarch64 Neoverse-V2 CXL mapping: /dev/mem at physical base 0x800000000 (same CXL window) Software: torch CPU + NO_GPU_EXT, LMCache Service on ARM Cores SHARED FABRIC CXL window: 32 GiB shared between host and ARM Inference engine: vLLM 0.22 + LMCache (Custom CXL connector) Connector: LMCacheMPConnector (Custom CXL transfer context) vLLM INFERENCE ENGINE CONFIGURATION CORE Model: Meta Llama-3.1-8B-Instruct BF16 Inference engine: vLLM (--enforce- eager) CONTEXT & ATTENTION Max context: --max-model-len 80000 tokens Attention: FlashAttention (Grouped Query Attention, 32:8) Chunk size: 512 tokens KV & MEMORY GPU KV budget: --gpu-memory- utilization 0.26 — small, so KV overflows GPU and forces offload Prefix caching: --no-enable-prefix- caching (LMCache manages caching) KV connector role: kv_both (store + retrieve) External memory: 32 GB (offload) © 2026 Marvell. All rights reserved. Source: Marvell