PDF 原檔：報告_OCP2026_SK海力士_運算共享記憶體池_20261015_original.pdf

講者公開簡報（OCP 官方 Recorded Content 頁提供下載）。文字以 PyMuPDF 逐頁抽取，### p<n> 為頁碼；圖表內容見原檔。

圖片清單（已驗證 2026-10-19）

原始內容

p1

Jongryool Kim Senior Director AI System Infra., Memory System Research SK hynix, Inc. Agentic AI Serving System with Computational Shared Memory Pool [FTI Workshop]

p2

This Talk • KV cache sharing to improve GPU utilization • KV cache selection offloading to a computational memory pool • Token sharing to reduce CPU load • Tokenization offloading to a computational memory/memory pool to mitigate CPU resource constraints

p3

• Memory pooling: Mitigate memory stranding by allocating CXL memory based on memory usage for each node • Memory sharing: Remove data transfer overhead and data duplication by sharing data objects between nodes Memory Pool Memory Pooling Memory Sharing

p4

Multi-rack Disaggregated Memory System • Memory-centric AI Rack System (MARS): A disaggregated rack-scale memory system that coordinates compute and memory as a shared infrastructure Inter Compute RACK and Memory RACK : RDMA In RACK : CXL Network Switch GPU Server GPU Server GPU Server ••• Memory Pool (CXL Memory) Network Switch GPU Server GPU Server GPU Server ••• Memory Pool (CXL Memory) Network Switch CPU Server CPU Server CPU Server ••• Memory Pool (CXL Memory) Network Switch ••• Memory Pool (CXL/CXL Hybrid Memory) GPU Rack GPU Rack CPU Rack Memory Rack RDMA CXL Memory Pool (CXL/CXL Hybrid Memory) Memory Pool (CXL/CXL Hybrid Memory) Memory Pool (CXL/CXL Hybrid Memory) ••• ••• KV cache sharing Token Sharing

p5

KV cache sharing for GPU

p6

• KV cache transfer (sharing) and reuse over pooled memory improved performance in LLM Inference • Direct fast reuse of KV cache stored in pooled memory during the sharing process whenever needed. • Bypasses secondary save-and-load operations, drastically cutting down data path latency KV Cache Sharing Server Server Pooled Memory (Sharing) 4 Server Server Pooled Memory (Sharing + Reuse) Server KV Cache 3 2 1 Prefill Decoding GPU 4 3 2 1 GPU Prefill Decoding GPU 4 3 2 1 GPU 4 3 2 1 4 KV Cache 3 2 1 Prefill GPU 1 2 3 4 … 7 6 5 4 3 2 1 … 7 6 5 Cache Reuse Mem-copy Mem-copy Mem-copy Mem-copy Mem-copy KV Cache Sharing KV Cache Reusing

p7

• Developed and Demonstrated a Pooled Memory-Based Distributed LLM Serving System Dynamo LLM Serving with CXL Pooled Memory (@ FMS 2025, OCP 2025, SC 2025) LLM Worker LLM Worker LLM Worker LLM Worker Processor Frontend Client Ethernet CXL Memory Pool Frontend/Processor/LLM Worker LLM Worker Niagara 2.0 LLM Worker [ FMS 2025 ] Dynamo LLM Serving System with CXL Memory Pool

p8

• Release GPU HBM immediately by offloading KV cache to pooled memory • Accelerates GPU HBM reclamation by offloading data to pooled memory immediately. • Maximizing Prefill hardware utilization. Fast HBM Reclamation KV cache KV cache KV cache KV cache KV cache GPU

5 KV

cache Prefill Server N I C GPU

1 KV

Cache Decode Server N I C The prefill GPU cannot release its memory if the decode GPU memory becomes full.

7 KV

cache KV cache KV cache KV cache GPU Prefill Server C X L GPU Decode Server C X L KV cache Memory Pool KV cache KV cache KV cache

1 KV

cache

6 KV

cache By offloading to pooled memory, the prefill GPU can conti nuously process requests

6 KV

cache OOM OOM OOM Network based KV Cache Sharing Pooled Memory based KV Cache Sharing

p9

• Avoid bandwidth (BW) contention across HBM, PCIe, and Network paths for KV cache Store/Reuse • Avoids dual DRAM/NIC data paths, resolving the BW contention seen in LM-Cache/Mooncake-like setups KV Cache Store/Reuse without BW Contention DRAM GPU KV KV Prefill Instance N I C DRAM GPU KV Decode Instance N I C RDMA KV Cache Store KV Put DRAM GPU KV KV Prefill Instance N I C DRAM GPU KV Decode Instance N I C RDMA KV Cache Store Put KV Cache Store RDMA Disaggregated Memory Pool GPU KV Prefill Instance C X L GPU KV Decode Instance C X L KV DRAM KV Cache Store Interconnect BW Contention between local offload and remote delivery Interconnect BW Contention between local offload and remote delivery between two remote deliveries No Interconnect BW Contention In-Server KV Cache Store Distributed KV Cache Store Pooled Memory KV Sharing

p10

• LLM Serving using CXL Pooled Memory can improve the LLM serving performance • 1.67× TTFT improvement over LM-Cache, 1.9x higher peak good put over NIXL (ALL recomputation) LLM Serving Performance Memory Pool for a Compute Rack Memory Pool for a Compute Rack

p11

KV cache selection

p12

• Offload KV cache selection for sparse attention to pooled memory and load only selected KV cache. → can improve the decoding latency KV Cache Selection Offloading to Pooled Memory Prefill Server GPU-HBM Context User question HBM KV 1 in KV 2 GPUs KV 3 provides KV 4 what KV 5 benefits? KV 6 Decode Server GPU-HBM Selected KV Answer HBM KV 1 provides KV 4 what KV 5 benefits? KV 6 speed Full KV retained in pooled memory HBM KV 1 in KV 2 GPUs KV 3 provides KV 4 what KV 5 benefits? KV 6 Gray entries remain stored for future turns (2) RocketKV Stage 1 SnapKV-style KV eviction (avg. x30 reduce) (1) Store full KV (3) Read selected KV only Decoding Prefill Eviction

p13

• Evicting KV entries can lead to context loss unless they are recomputed from the conversation history. • To preserve full context without re-computation, the KV cache for the entire conversation must be retained. Why do we store the entire KV cache? Turn 1: What benefit? Turn 2: Which devices? Pooled Memory / DRAM: full prompt KV retained 1 2 3 4 5 6 1 2 3 4 5 6 Turn 1 prompt KV Retained Turn 1 prompt KV New Turn 2 question Which 7 devices? 8 Pool-side Stage 1 selection Pool-side Stage 1 re-selection Transfer selected KV Transfer selected KV Decode GPU-HBM: selected copies HBM 1 provides 4 what 5 benefits? 6 HBM 1 GPUs 3 Which 7 devices? 8 Answer: speed Answer: GPUs Full KV survives permanent selection HBM in GPUs provides what benefits? HBM in GPUs provides what benefits?

p14

• Transfer only selected KV cache entries to the decoding GPU. • Reduce the amount of data transferred and shorten KV cache transfer time. • Enable decoding to start sooner, lowering Time to First Token (TTFT). Selective KV Cache Transfer Reduces Latency Full KV baseline Compression offloading Decode GPU Decode GPU HBM capacity is sufficient HBM capacity is sufficient 1 2 3 4 5 6 HBM 1 provides 4 what 5 benefits? 6 6 resident KV entries 4 resident KV entries (avg. x30 reduce) Decode Attention Ops Decode Attention Ops Read full KV Read selected KV Larger KV memory traffic Smaller KV memory traffic Smaller KV memory traffic without capacity pressure HBM in GPUs provides what benefits?

p15

• Transferring only selected KV cache entries reduces the HBM footprint per request on the decoding GPU. • This alleviates HBM capacity constraints that can otherwise block requests from joining the decoding batch. • Requests can enter the decoding batch sooner, reducing queuing delay → low e2e latency Selective KV Caching Reduces Queuing Delay Prefill Server GPU-HBM New request’s full KV cache 1 2 3 4 5 6 Decode Server GPU-HBM A B C … Existing requests Only 4 KV slots available Pooled Memory (Sharing + Reuse) 1 2 3 4 5 6 Full KV retained (1) Store full KV unchanged Full copy needs 6 slots HBM in GPUs provides what benefits? HBM in GPUs provides what benefits?

p16

• General Cores only • High Performance & Highly Flexible PNM • CXL.mem Command Signal • Fast Offloading • Finer-grained Command Offloading • Full Stack SW environment • Attention API/Vector Search Library for AI • Programming Model • Device Runtime CMM-Ax 3.0 CMM-Ax Form Factor FH3/4L AIC Full Height ¾ Length Add-in-Card Interconnect CXL 2.0 / Device type 3 / PCIe 5.0 x16 Memory* DDR5-6400 4-ch / 512GB / 200GB/s Controller 16 ARM Neoverse V2 cores at 3.0GHz

p17

• Overlap the next layer’s prefill computation with KV cache transfer and pool-side compression. • CMM-Ax runs RocketKV Stage 1 and selects KV indices for each layer. Pipeline Optimization Layer L+1 compute overlaps layer L offloading Time Prefill GPU Transfer to CXL pooled memory CMM-Ax Stage 1 (scoring) Prefill layer L Prefill layer L+1 Prefill layer L+2 … Full KV cache (L) Last query block KV cache scoring Layer (L) Later: Decode fetches KV with the selected indices CXL pool 1 2 3 4 5 6 Selected indices {1, 4, 5, 6} Decode GPU HBM: selected KV cache 1 4 5 6 Index-guided KV fetch Full KV stays in the pool Full KV cache (L+1) Last query block KV cache scoring Layer (L+1) HBM in GPUs provides what benefits?

p18

Results Opus-4.5 trace runtime comparison ARCEE trace runtime comparison • Overlap the next layer’s prefill computation with KV cache transfer and pool-side compression. (with Structera CXL) • CMM-Ax runs RocketKV Stage 1 and selects KV indices for each layer. • 1.4x TTFT improvement and 2.29x E2E Latency improvement over Mooncake (Arcee trace) Memory Pool w/ KV Selection Memory Pool Mooncake Memory Pool w/ KV Selection Memory Pool Mooncake Memory Pool w/ KV Selection Memory Pool Mooncake Memory Pool w/ KV Selection Memory Pool Mooncake

p19

Token sharing & Offloading for CPU

p20

• 25.6K input tokens, prefix KV cache reuse 96%*. vLLM text tokenization latency removed Time to First Token (lower is better) ≈345 ms ≈215 ms 0 100 200 300 400 TTFT (ms) ≈ 38% lower TTFT * Coding-agent workloads have reported prefix KV cache hit rate of ≈ 96%. Text Tokenization Cost

p21

• Removing image tokenization improves serving throughput and latency. vLLM Image tokenization removed Time to First Token (lower is better) 1.80× 1.76× 0 200 400 600 800 Light load Medium load TTFT (ms) Image Tokenization Cost

p22

• Computational CXL reuses prefix token IDs and tokenizes only new text. Input prompt Repeated prefix New text Token IDs for vLLM Reused prefix IDs New IDs Computational CXL memory Arm cores (2) Search saved prefix (3) Tokenize only new text Memory Saved prefix IDs (1) Send full prompt Reuse token IDs Save token IDs (4) Return full token IDs Token Sharing in Computational CXL

p23

• 25.6K input tokens, prefix KV cache reuse 96%*. vLLM vLLM + token sharing in C-CXL Tokenization Time Time to First Token ≈130 ms ≈6 ms 0 50 100 150 Latency (ms) ≈345 ms ≈221 ms 0 100 200 300 400 TTFT (ms) ≈ 95% lower ≈ 36% lower 2) Token Sharing in Computational CXL

p24

Learn More about Memory Pool • SK hynix booth : C39

p25

（本頁無可抽取文字，內容在圖中）