video_OCP2026_Samsung_高擴展記憶體_20261015

The moment a user enters a prompt, the AI inference journey begins. Powered by advanced semiconductors at every step. The prompt flows through server DRAM to the CPU, where it is converted into [music] tokens. As a gentle AI runs multiple tasks, token generation and memory traffic search, making memory bandwidth critical to system performance. To reuse information across massive context windows, GPUs build a KV cache and keep it in HBM. Scaling beyond HBM2 DRAM, CXL, and SSD as the cache grows. At scale, output tokens are generated and converted back into text, delivering responses in the blink of an eye. Behind every AI response, Samsung's memory [music] keeps AI workload moving at scale. >> Okay. Good afternoon, everyone. We are back at OCP 2026. Many of you probably used the AI just like in this video when preparing for your trip here today. Right? Okay, getting on AI responses seems effortless on the surface, but the underlying infrastructure behind it is anything but simple. I am Woojin Shim, leading memory application engineering at Samsung Electronics. Today, my topic is a scaling innovation for the AI era. How to keep your compute running at full throttle with highly scalable memory. Every token moves bytes and the far more tokens are being served by the industry. Multiply those two numbers, we get the architectural challenge. On the left, gigabytes read from memory per generated token. This year, GLM 5.2, 1 million context, hits 55 gigabytes per token. This is a six times explosion in memory rates in 2 years. In 2024 and the 2025, the model weights dominated memory traffic. This year, that changed completely. Context, KV cache explodes. KV cache compression is working. Thanks to advanced attention techniques like MLA, KV cache size per token decreased. However, context expansion is completely outrunning every compression technique we have. On the right, KV cache created per month in exabytes in a single hyperscaler. Token volumes disclosed at Google I/O keynotes. This year, it is 140 exabytes. That is a 60 times explosion in KV cache size in 2 years. 400 concurrent 1 million token session generated 18 terabytes of KV cache. A variable in 200 MVR 2072 rack holds 20 terabytes of HBM. Please note that 400 sessions consume an entire rack. That is the demand side. Now, let's look at what the hardware actually delivered. Capacity grew at every tier, but none of it grew enough. Here is the deployed capacity per GPU from 2024 to this year. Comparing Hopper with Rubin. First, HBM for speed of two times. Second, memory expansion such as high-density DDR5 RDIM and LPDDR So-DIMM 2 of three times. Third, context storage for persistence. Please note that this bar is split. Local SSD doubled per GPU. On top of it, CMX context memory extension, 16 TB per GPU. CMX is a new flash-based KV cache tier defined by the platform vendor. This proves the context storage problem is real. Together with the local SSD, context storage grew six times. Every grew Every tier grew two to six times. As I mentioned in the previous slide, total bytes read per generated token grew six times, and KV cache created per month grew 60 times. To close this gap, Samsung approaches the system across three architectural pillars: speed, memory expansion, and the persistence. Pillar one, speed. Decoder is a fundamentally memory bandwidth bound. So, instead of asking expensive compute to wait, we move compute to memory to accelerate the decode where the tokens are made. On the scale-up GPU side, we drive two parallel axes. One is the HBM generation roadmap. HBM 3E, HBM 4, 4E through HBM 5 for higher bandwidth. The second axis goes beyond HBM roadmap. In order to move compute close to memory, custom HBM integrates customer logic into the base die. ZHBM uses a direct bonding right on top of the GPU by eliminating the conventional silicon interposer and the micro bump stacks. Compared to HBM 5, ZHBM delivers four to eight times the bandwidth and three times the energy efficiency. On the scale of XPU side, we bring processing in memory, LPDDR PIM. Compared to LPDDR, LPDDR PIM delivers eight times the effective bandwidth and three times the TPS performance. We measured the system-level performance on HGI accelerator SOC. Gemini that dominates the decoder workload are fully loaded to LPDDR PIM. Then, six times faster Gemini operation on the PIM die than on host AP execution. Finally, we achieved three times the TPS. We designed the base die, the inter layer, and the controller that one makes these numbers possible. This time, let's see how decoder throughput scales with memory bandwidth. We measured it on VLLM across HBM bandwidth settings on DGX B200 in Samsung's own lab. 17% more HBM bandwidth delivers 12 to 15% faster decode. We tested across 100 to 400 concurrent users with the Llama 3 prefill to decode one-to-one disaggregation conditions. So, what does this prove? First, token generation at the decoder stage is bandwidth bound, not computer bound. Second, HBM bandwidth converts into tokens almost linearly. And third, every gigabyte per second added is a usual TPS you can charge extra for. Next pillar, pillar two, memory expansion. Here is the harsh reality on the memory tier. Long context session generates massive active working set. When it does not fit in HBM, DDR5 R-DIMM, or LPDDR5 CAMM2, the GPU is a forced to recompute it. Recomputation is a waste of expensive GPU computer resource you already paid for. Which is why we say computer stalls when memory capacity hits a wall. So, Samsung breaks through the memory capacity wall with the CXL MD and the CXL memory pooling. The first is a scale up with the CXL MD. Our trajectory moves from MD220 to MD310, which is available for customer POC. Next is MD320 LPDDR based, currently in development. Through compression and default allocation, MD320 targets 4 terabytes of effective capacity and the 20 times the energy efficiency of MD310. We are also integrating full CXL 3.x features, including dynamic capacity device and a uh trusted security protocol. The second is a scale-out CXL memory pool. It delivers 10 to 80 TB of capacity and 782 GB per second of bandwidth. Why does memory pooling matter? Because node attached capacity can now move to the instance that needs it. Pooling enables hot KV cache and hot MOX ports to move to the instance that needs them. So, Samsung is a proactively building CXL attached to memory pooling open ecosystem with a Marvell, Liquid, and the LM cache open source software with a TensorMesh. Adding capacity is one thing. Making it dynamic and allocatable is another. And it is measurable. We measured on V LLM with a CMMD expansion and a CXL memory pooling. On the left, CMMD expansion improves TPS by 5.6 times without adding any GPU. Please note, 5.6 times more tokens without adding any GPU. The right side is the one I care more about. CXL memory pooling is tested and the elastic capacity allocation reduces TTFT by 38% and increases throughput by 3.2 times. Please note, the same amount of silicon. The difference is elastic capacity allocation. Adding capacity is easy half. Making it dynamic and allocatable is the other half that cuts the TTFT. Pillar three, persistence. In the agentic storage joins the memory hierarchy. Nvidia defined this tier with the CMX. The question is, what survives inside it? Samsung brings two scale-up SSD breakthroughs. First, our Gen 6 SSD PM1763 delivers a 28 GB per second bandwidth to minimize storage bottleneck in AI and HPC. Second, our E3.S form factor QLC SSD, PM1773, 256 TB capacity, delivers a maximum density for AI data lakes and hyperscale. But, here is the SSD endurance problem. KV cache and AI agent session workloads demand DWPD of 3+. That means a dedicated KV cache flash tier is a right amplification worst case. QLC alone cannot survive that traffic. So, we have two answers. First is FDP. It cuts WAF from three or four down to one. And the mixed media SSD, it places hot writes into high endurance SSD model regions. These two, FDP and the mixed mode SSD, are to improve SSD endurance. Samsung drove over FDP, the open standard this tier runs on, into the Linux corner. I will show how FDP performs under real KV cache writes in the next slide. On the scale-out SSD storage side, Samsung is collaborating with several software-defined storage partners like Hammerspace, Weka, Cloudian, and Scality. Together with a hammer space, we built a 1.8 petabyte storage cluster and the ranked number 18 on the IO500 10 node production list at supercomputing 2025. This proves that standard Linux and the Samsung enterprise NVMe SSD can achieve supercomputing class IO performance. As I mentioned before, FDPC cures SSD fleet endurance under heavy KV cache rights, mix of sequential appends and random invalidations. This is the worst case for a conventional FTL and the triggers constant garbage collection. We measured the WAFF under multi-tenancy with mixed workload on our Lustre-based distributed storage. A conventional SSD hit a painful WAFF of 3.4. With the Samsung S FDP, we eliminated garbage collection and the collapsed WAFF from 3.4 down to 1.1. So, what does this prove? First, FDP extends SSD replacement cycles. Second, FDP eliminates tail latency, which is important for you to hold your SLO. Third, WAFF is not a drive metric. It is how many drives you need to buy over the life of the fleet to generate the tokens in the AI era. Now, we bring one roadmap across the all three tiers. First, speed. Driving memory bandwidth from HBM3E through HBM5. And extending beyond with a custom HBM and the ZHBM. Second, memory expansion, delivering memory capacity with high-density DDR5 R-DIMM or LPDDR4 CAMM2, and elasticity by scaling from DDR5 CMM-D to LPDDR-based CMM-D. Third, persistence, expanding storage bandwidth, capacity, and the endurance. PCIe Gen6 for bandwidth, QLC for capacity, and the FDP for endurance. Three tiers, one timeline. That is the co-design surface for scaling innovation in the AI era. Co-design starts before the spec is frozen. Samsung is open to provide early silicon, real measurements, and the sit at the design table. To make the co-design real, we are operating Samsung Max Lab, memory application experience lab, as an official OCP experience center. It is an open collaboration platform in the real data center environment. It runs the Samsung's latest technologies, including pre-production silicon, such as PIM, CMM-D, and the Gen6 SSD. And the Samsung Max Lab supports cluster-level evaluation running 24 server racks. In parallel, hardware is useless without software. Samsung GOST, Global Open Ecosystem Team, is the host of open open-source open ecosystem activities for memory solutions. It brings it brings together maintainers and contributors to numerous open-source projects, such as FDP, large block IOU, AISIO, and the LM Cache. So, here is our invitation. Bring your requirement, benchmark, or an engineer. We will bring silicon you can't buy it. Okay, so this is a how we we make your computer running at full throttle every day with the highly scalable memory. So, Samsung is open to collaborate with partners and customers. Thank you.