報告_OCP2026_SK海力士_MARS記憶體中心機櫃_20261015

PDF 原檔：報告_OCP2026_SK海力士_MARS記憶體中心機櫃_20261015_original.pdf

講者公開簡報（OCP 官方 Recorded Content 頁提供下載）。文字以 PyMuPDF 逐頁抽取，### p<n> 為頁碼；圖表內容見原檔。

圖片清單（已驗證 2026-10-19）

原始內容

p1

Hoshik Kim Senior Vice President and Fellow Memory Systems Research SK hynix, Inc. Introduction to Memory-centric AI Rack System (MARS) [Executive Sessions]

p2

Memory Wall Challenge • Memory demands in AI continues to grow rapidly, while scaling GPU memory faces physical and economic constraints • Unlocking the next level of AI performance requires rethinking how data is stored, accessed, and shared • Compute systems must be re-architected around disaggregated memory/SSD system Orchestrator Agent Researcher Agent Coding Agent Researcher Agent Reviewer Agent Complex Objective Output Accumulated Context (KV Cache) Explosion AI Researcher Agent Workflow → Exponential KV cache Growth 10+ Multi-turn Agent-to-Agent Context Transfer Turn 3 Turn 2 Turn 1 · · · HBM GPU Server DRAM CPU HBM GPU SSD Server Disaggregated Memory and Storage CPU CXL Memory SSD • Model Size Growing • Context Length Growing • Multi-Turn Conversation • More Concurrent Users and Sessions

p3

A Paradigm Shift: From Processor-centric to Memory-centric • Processor-centric: CPUs/GPUs perform computation while memory acts merely as a subordinate support • Memory-centric: A large, shared memory serves multiple compute units, enabling more flexible access to data CPU CPU CPU GPU GPU GPU GPU GPU Network Processor-centric Memory-centric Memory CPU GPU CPU GPU CPU GPU GPU GPU

p4

Problems in Processor-centric Architecture • Idle usage on one server cannot easily relieve pressure on another Network Server • • • A (Memory Pressure) Server B Free Server • • • Server D Free C Free • Enough memory in the cluster. Not enough where it is needed. - Memory exhausted in some nodes, idle in others • Resource Imbalance - Must buy entire new server node to add memory • Rigid Scalability - GPU over-provisioning for memory access drives up costs • High TCO - Slow network transfers and redundant copying between nodes • Data Silos & Latency

p5

Breaking Server Limits with Disaggregated Memory • Memory disaggregation decouples memory from traditional CPU/GPU server constraints, creating a dedicated, massive "Memory Pool.“ RDMA Network Compute Server • • • • • • Free A • From per-server headroom to pool-level efficiency. - Use pooled capacity to balance demand across servers • Higher Memory Utilization - Add memory without adding CPU/GPUs • Independent Scaling - Assign and reclaim capacity as demand changes • Flexible Capacity Allocation - Reduce excess provisioning and unnecessary server purchases • Potential TCO Savings Memory Server Compute Server Memory Server B C D

p6

High-Speed Data Sharing via Memory Pools • Multiple AI accelerators (GPU/NPU) directly share large LLM models/KV caches or pipeline results within the pool • Reduce communication overhead: Replace explicit server-to-server data transfers with memory access over a CXL fabric • Streamline AI pipelines: One accelerator writes its output to shared memory, the next accesses that data without an additional application- level transfer Network based KV Cache Sharing Pooled Memory based KV Cache Sharing LLM Model KV Cache GPU Server AGENT #2 NIC AGENT #1 LLM Model KV Cache GPU Server AGENT #3 NIC LLM Model KV Cache GPU Server NIC Network Switch LLM Model KV Cache GPU Server AGENT #2 AGENT #1 LLM Model KV Cache GPU Server AGENT #3 LLM Model KV Cache GPU Server Pooled Memory KV Cache

p7

Multi-rack Disaggregated Memory System • Memory-centric AI Rack System (MARS): A disaggregated rack-scale memory system that coordinates compute and memory as a shared infrastructure Inter Compute RACK and Memory RACK : RDMA In RACK : CXL Network Switch GPU Server GPU Server GPU Server ••• Memory Pool (CXL Memory) Network Switch GPU Server GPU Server GPU Server ••• Memory Pool (CXL Memory) Network Switch CPU Server CPU Server CPU Server ••• Memory Pool (CXL Memory) Network Switch ••• Memory Pool (CXL/CXL Hybrid Memory) GPU Rack GPU Rack CPU Rack Memory Rack RDMA CXL Memory Pool (CXL/CXL Hybrid Memory) Memory Pool (CXL/CXL Hybrid Memory) Memory Pool (CXL/CXL Hybrid Memory) ••• •••

p8

MARS Software Stack dThread Inter-Node Lock Memory Copy API (GPU-Mem Pool) Memory Allocator RPC CCL MPI Streaming GNN Storage GNN Tool NATS In-Memory Store (KV, FS, Object Intef.) Metadata Server KV cache Store KV cache Tiering Graph RAG • We are co-developing a variety of software solutions with ecosystem partners to enable MARS systems

p9

Breaking AI’s 3 Bottlenecks: Capacity, Latency, and Complexity • Memory-centric AI rack system structurally resolves hardware limitation facing next-generation AI inferencing services Extend available memory beyond GPU memory through a shared memory pool Latency Capacity Data Management Complexity Boosts inter-node data streaming, enabling real-time interaction Merges partitioned data into a memory pool for low latency data retrieval

p10

Multi-Turn AI Inference and KV Cache Acceleration • Multi-turn agentic AI services cause KV cache to explode, exhausting GPU memory and stalling systems • Fully manage KV cache in a memory pool can increase system throughput and provide reliable Time-to-First-Token (TTFT) Memory Rack for Multiple Compute Racks Network Switch ••• Memory Pool (CXL/CXL Hybrid Memory) Memory Rack Memory Pool (CXL/CXL Hybrid Memory) Memory Pool (CXL/CXL Hybrid Memory) Network Switch GPU Server GPU Server ••• GPU Rack GPU Server KV Cache KV Cache Hybrid Memory NAND CXL IF DRA M Cache CTRL Time-to-First-Token Recomputation Remote SSD Remote CXL DDR Remote CXL Hybrid 3.5–5.2× improvement in TTFT over re-computation 400 Time to first token (S) 300 200 100 0 Re-computation Remote SSD Remote CXL DDR 478.11 259.03 122.90 Tail Remote CXL Hybrid 136.95 80.76 33.56 15.45 21.92 Average

p11

Low-Latency Voice Interaction for Personal AI Agent Service • Traditional voice pipelines (Audio Encoder → LLM → Text To Speech (TTS)) suffer from latency due to data copying and network overhead between models, causing unnatural pauses • All AI models in the voice pipeline operate on the same Shared Memory region within the Disaggregated Memory Pool Speech-to-Text Thinker (LLM) Text-to-Speech Visual Motion Text chunks Audio codec chunks Thinker Thinker Code2Wav Motion + Inter Model Output Streaming using Memory Pool Audio Text chunks Audio codec chunks Audio waveform chunks CXL Memory Pool 1.46× TTFA and 1.41× E2E latency improvement over RDMA Memory Pool RDMA TTFA Speedup 1.46x E2E Speedup 1.41x Speech-to-Text Thinker (LLM) Text-to-Speech Speech-to-Text Thinker (LLM) Text-to-Speech 0.21 s 2.35 s E2E 8.08s TTFA 0.46s TTFA 0.67s E2E 5.73s First Audio Chunk First Audio Chunk Increased GPU Util. Improved Quality of Service (QoS) Inter Model Output Streaming using Memory Pool

p12

Unified Graph Sampling for Graph Neural Network Training • A trillion-scale graph data are partitioned across multiple servers, leading to frequent communication during sampling • Offloading graph/feature data to memory pool can eliminate sampling synchronization and improve data read latency 0 1 2 3 4 5 0 1 2 4 3 5 Graph Structure Feature Table Partitioning Server GPU 0 1 3 4 0 1 2 Feature Gathering 0 Node 0 Sampling Server GPU 0 2 4 5 3 4 5 Feature Gathering 4 Node 0 Sampling RDMA Many RDMA based all-to-all synchronizations Server GPU Feature Gathering 0 Node 0 Sampling Server GPU Feature Gathering 4 Node 0 Sampling Constant latency direct access Memory Pool 0 1 2 4 3 5 0 1 2 3 4 5 Neighbor Sampling Remote Feature Gathering Distributed GNN Training using RDMA Distributed GNN Training using Memory Pool 17.57× Sampling time improvement over RDMA

p13

Toward a New Foundation for AI Infrastructure • Memory-Centric AI Rack Systems (MARS) structurally resolve hardware limitations facing next-gen AI services • Capacity expansion for multi-turn AI inference (Multi-Turn conversation, Multi-Turn Coding Agent) • Low latency for personal AI agent • Unified data management for Graph Neural Network Training • MARS are no longer an option but a requirement for operating massive AI systems • MARS supporting customized pilot tests for clients and expanding the ecosystem

p14

One More Thing …

p15

HBF Extends HBM-class Bandwidth to Flash-scale Capacity • A New AI memory layer between HBM and eSSD A New AI Memory Tier Between HBM and eSSD Capacity Scaling Extend beyond HBM capacity limits Bandwidth at Scale Maintain high-speed access as capacity grows Data Movement Reduction Keep more AI data closer to compute Energy Efficiency & Bandwidth Capacity 0.5x 14x 3.7x HBF HBM

1x 1x 1x <0.1x 500x 6.9x eSSD Bandwidth Capacity pJ/b HBF: Bridging the gap between HBM and eSSD

p16

HBF Breaks the Memory Capacity Wall • HBF provides x1/2 Bandwidth and x10 Capacity of HBM to break the Capacity Wall HBF 1M Tokens/user 15M 10M 5M 19M 9M 1.2M 1 8 16 32 64 128 256 DeepSeek R1 671B (8 cards, Expert parallelism 8) Llama 3 70B (1 card) HBF Base Batch *Base card capacity is 288GB and HBF card 3TB 1/2x Bandwidth, 10x Capacity of HBM Breaking the Capacity Wall

p17

Building the Standard and Ecosystem for HBF Now v 0.7 Public Release HBF Product HBF Introduction @SanDisk SanDisk HBF Introduction MOU Signed Consortium Kick-off w/ Spec α Public Release Spec 1.0 Release HBF Cube Available TBD ‘25.2 ‘25.8 ‘26.2 ‘26.7 ‘27.1 (Expected) ‘28.1Q MOU Consortium Kickoff HBF Cube Collaboration with partners

p18

Thank You!