PDF 原檔：報告_OCP2026_SK海力士_儲存作為LLM效能層_20261015_original.pdf

講者公開簡報（OCP 官方 Recorded Content 頁提供下載）。文字以 PyMuPDF 逐頁抽取，### p<n> 為頁碼；圖表內容見原檔。

圖片清單（已驗證 2026-10-19）

原始內容

p1

Joohee Lee Research Engineer, SK hynix Donguk Moon Director, SK hynix Storage as a Performance Tier for LLM Serving STORAGE

p2

Prompt Caching: Saving Computation KV Compute Overhead Prompt Caching Example Example from SqueezeBits Blog Request #1 Request #2 Request #3 A D A G A L Prefill Prefill (Cached) Decode B B B C F K E H M I N J • Caches KV states of shared prompt prefixes (e.g., system instructions, context, examples), to avoid redundant prefill computation. • Reuses cached KV across requests and conversation turns. →Lower GPU compute & TTFT Request #1 A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user’s questions. User: Hello! Request #2 A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user’s questions. User: How are you? 600 500 400 300 200 100 0 10 5000 10000 15000 Sequence Length Latency (ms) Feed-forward KV Compute

p3

Hierarchical Prompt Cache • Production LLM services retain cached prefixes to maximize reuse. - Anthropic provides 5-minute and 1-hour cache Time-to-Live (TTL) options. - OpenAI supports extended prompt caching with a TTL of up to 24 hours. - Longer TTL →More reusable KV states →Larger KV working set. The Growing KV Cache Working Set Working Set KV caches are primarily stored in GPU HBM and host DRAM. Excess KV caches are offloaded to SSDs. HBM/DRAM capacity is insufficient for 10-minute KV cache retention in chat workloads. For Qwen3-14B (BF16), a 10-minute TTL requires up to ~781GB of KV cache capacity per GPU. HBM: 140 GB (Model Size: 50GB) DRAM: 375 GB SSD: 1~10 TB Offloading KV Overflow to DRAM Offloading KV Overflow to SSD NVIDIA B200 Server 781GB 348GB 1000 800 600 400 200 0 0 50 Time Window KV Cache Capacity (GB) SSD DRAM HBM *Qwen-Bailian Anonymous Dataset

p4

• Longer cache retention improves reuse opportunities, but rapidly expands the KV working set beyond HBM/DRAM capacity. • Production LLM services are moving KV caches to storage. Scaling Prompt Caching Beyond Memory Cache On chip DRAM Directly attached Local SSD Locally attached Remote Storage Size: 1x Latency: 1-40ns TTL: Seconds Size: 10x Latency: 80-140ns TTL: Several Minutes Size: 100x Latency: 10-40us TTL: Tens of Minutes Size: 10,000x Latency: 3-10ms TTL: Hours Increasing bandwidth Increasing TTL, latency, and capacity Server Memory Hierarchy - OpenAI: GPU + CPU RAM + Storage - DeepSeek: GPU + Storage - Google Cloud: GPU + CPU RAM + Storage

p5

• Without prompt cache reuse, prefill is GPU compute-bound due to full KV re-computation. Thus, TTFT scales almost linearly with GPU compute performance. • With prompt cache reuse, prefill retrieves KV data from outside the GPU and decodes only one token, making TTFT dependent on link bandwidth (for KV transfer) and GPU memory bandwidth (for decoding). Thus, improving these bandwidths yields nearly inversely proportional TTFT gains. Prefill Performance 8 6 4 2 0 A100 H100 GH200 TTFT (sec) GPUs A100 H100 GH200 FP16 (TFLOPS) 624 1,513 1,979 0.4 0.3 0.2 0.1 0 A100 H100 GH200 TTFT (sec) GPUs A100 H100 GH200 Link BW (GB/s) 32 64 450 1.9 2 4 Mem BW (TB/s)

p6

• When using SSDs, achieving peak performance requires exploiting their internal parallelism – across channels, packages, dies, and planes – so that KV traffic can be served with maximal bandwidth. SSD Parallelism GPU SM SM SM SM SM SM SM SM HBM SSD Controller Plane Plane Plane Plane Die Die Plane Plane Plane Plane Die Die Plane Plane Plane Plane Plane Plane Plane Plane Package Package Channel Single Request Multiple Request Interconnect (e.g., PCIe) SSDs

p7

Evaluation Setup LMCache v0.3.1 Microbenchmark, Macrobenchmark (ShareGPT) NVIDIA H100 GPU vLLM v0.9.1 Benchmark Qwen3-14B • System Configuration - Serving Framework: vLLM v0.9.1 + LMCache v0.3.1 - Model: Qwen3-14B (BF16) - GPU: NVIDIA H100 PCIe (PCIe Gen5 x16) - Storage: NVMe SSD (PCIe Gen5) - SSD I/O Optimization: Linux io_uring-based asynchronous, multi- threaded I/O • Benchmark Configuration • Microbenchmark - Synthetic prompt cache traffic with 100% cache hit - Isolates the impact of SSD internal & external parallelism - Measures the best achievable TTFT for cached prefill • Macrobenchmark - ShareGPT-based realistic multi-turn workload - Request rate (QPS) sweep to evaluate serving performance under varying loads - Equal cache capacity for DRAM and SSD

p8

SSD Performance External Parallelism Internal Parallelism • [Internal Parallelism] Key I/O parameters that impact SSD performance include request size (block size), I/O depth, and the number of issuing threads. • [External Parallelism] GPUs operate over a PCIe x16 interface while each SSD occupies a PCIe x4 link. 1 0 0.2 5 0.5 0.7 5 IO Parameters File Size Normalized Throughput 1 0 0.2 5 0.5 0.7 5

of SSDs

Normalized Throughput

p9

Microbenchmark • Without any additional optimizations, scaling from 1 →4 SSDs (MD4) resulted in 1.26x~1.92x higher TTFT than the host-memory baseline. • With LMCache chunk size tuning and io_uring optimizations applied, TTFT approached the host-memory baseline, reaching 1.07x~1.14x of the memory performance. Prompt Caching Performance Macrobenchmark (ShareGPT) 3.5 3 2.5 2 1.5 1 0.5 0 1K 4K 16K 32K Prompt Length Normalized Latency MEM SSD×1 SSD×2 SSD×3 SSD×4 OPT 5 10 15 20 0 300 100 200 TTFT (sec) 0.1 (3%) 0.2 (5%) 0.3 (0%) 0.4 (-3%) 0.5 (2%) Throughput (K tokens/sec) Memory OPT (SSDs)

p10

• We define the endurance-bound capacity as the minimum SSD capacity required to sustain the expected daily write amount under given DWPD (Daily Writes Per Day) constraint. SSD Endurance Considerations 𝐶𝑏𝑜𝑢𝑛𝑑= 𝑡𝑜𝑡𝑎𝑙𝑤𝑟𝑖𝑡𝑒𝑠𝑝𝑒𝑟𝑑𝑎𝑦 𝑡ℎ𝑒𝑛𝑢𝑚𝑏𝑒𝑟𝑜𝑓𝑆𝑆𝐷𝑠× 𝐷𝑊𝑃𝐷 𝑇𝐵𝑊(𝑇𝑒𝑟𝑎𝑏𝑦𝑡𝑒𝑠𝑊𝑟𝑖𝑡𝑡𝑒𝑛) = 𝑃/𝐸𝐶𝑦𝑐𝑙𝑒𝑠×𝐶𝑎𝑝𝑎𝑐𝑖𝑡𝑦 𝑊𝐴𝐹(𝑊𝑟𝑖𝑡𝑒𝐴𝑚𝑝𝑙𝑖𝑓𝑖𝑐𝑎𝑡𝑖𝑜𝑛𝐹𝑎𝑐𝑡𝑜𝑟) 𝐷𝑊𝑃𝐷= 𝑇𝐵𝑊 𝐶𝑎𝑝𝑎𝑐𝑖𝑡𝑦×𝑊𝑎𝑟𝑟𝑎𝑛𝑡𝑦𝐷𝑎𝑦𝑠

p11

• SSD-backed prompt caching generates heavy write traffic from continuous KV cache writes. • With Qwen3-14B, write traffic can reach up to ~57.5TB/day per GPU. • Some enterprise SSDs cannot meet the required Drive Writes Per Day. • Endurance-aware SSD selection and design are essential for sustainable deployment. SSD Endurance Considerations 0.5 0 0.1 0.2 0.3 0.4 Cbound (TB) Daily Write Volume (TB) (a) SLC 0 100 25 50 75 0.29 25 0 5 10 15 20 Cbound (TB) Daily Write Volume (TB) (b) TLC 0 100 25 50 75 14.4 4.8 OP 50 0 10 20 30 40 Cbound (TB) Daily Write Volume (TB) (c) QLC 0 100 25 50 75 28.8 8.8 ZNS Daily Write Traffic (57.5TB) Minimum SSD Capacity To Satisfy Endurance Constraints Reduced Capacity Bounds Achieved by OP and ZNS

p12

Remaining Challenge: Non-uniform SSD I/O Latency • Even after SSD-aware optimization, per-chunk I/O latency remains variable. 0 40 30 20 10 1.5 0.5 1 IO Time (msec) Chunk ID Memory OPT (SSDs) - DRAM retrieval shows consistent per-chunk latency. - SSD retrieval exhibits latency fluctuations even for same-sized KV chunks. - Variability remains despite fully optimized SSD I/O

p13

Remaining Challenge: Irregular I/O Completion • The SSD I/O pipeline exhibit initial queue buildup and irregular request completions. - ⓐInitial Queue Buildup: I/O requests accumulate before the first completions arrive. - ⓑIrregular Request Completion: Queue drains at non-uniform intervals, even for same-sized requests. 100 80 60 40 20 0 0 1 2 3 IO Depth Timestamp (msec) ⓐⓑ 1.1 1.3 1.5 1.7 40 50 60

p14

Toward KV-aware Storage • Block-oriented operation and software I/O processing can introduce service-time variability. • KV-aware storage exposes key-value semantics to storage (e.g., KV SSDs), avoiding conventional block- address translation and I/O processing. • This can reduce retrieval overhead and improve service-time consistency. SSD Management KVSSD Management Host IO Stack User Request User Request Latency Overhead Reduced Parallelism Latency Reduction Improved Parallelism Block Request

p15

Toward Higher Storage Bandwidth • Limited system-level PCIe allocation to NVMe storage can constrain SSD bandwidth below the GPU interconnect capacity. • Scaling aggregate storage bandwidth requires sufficient PCIe resources, denser fabrics, and faster interfaces to keep pace with future GPUs. SSD CPU Switch GPU CPU Switch GPU Allocate More Bandwidth to SSDs Use Cost-effective, High-capacity SSDs SSD SSD

p16

Key Takeaways • Prompt caching shifts LLM prefill from compute-bound to data movement bound, making efficient KV transfer critical to TTFT. • SSD internal and external parallelism must be fully exploited to maximize effective SSD-to- GPU bandwidth. • With SSD-aware I/O optimization, SSD-backed prompt caching can achieve near-DRAM-class TTFT, even under realistic multi-turn workloads. • SSD endurance is a first-class deployment consideration; OP, ZNS, and FDP can substantially improve write endurance and reduce capacity requirements. • Remaining I/O variability reveals further optimization opportunities through tighter HW/SW co- design and KV-aware storage (e.g., KV SSDs). • SSD is not just a capacity tier – it can become a high-performance KV cache tier for scalable LLM serving.

p17

Call to Action • Problem to solve - Design AI serving systems with SSDs as a performance-critical cache tier, not just passive capacity. - Provision sufficient system-level I/O bandwidth for storage aggregate SSD bandwidth aligned with rapidly growing GPU interconnect bandwidth. - Explore KV-aware storage interfaces (e.g., SNIA Key Value Storage API) and HW/SW co-design for low- overhead, more predictable KV retrieval. • Feel free to contact us for any questions, discussion or collaboration - Project Wiki: https://github.com/skhynix/LMCache/wiki/Storage-as-a-Performance-Tier-for-LLM-Serving - E-mail: joohee.lee@sk.com, donguk.moon@sk.com • Potential OCP Collaboration - Evaluate storage technologies and interfaces such as FDP, ZNS, and KV SSDs for LLM KV cache workloads.

p18

Thank You!