video_OCP2026_Samsung_代理記憶體分層KVCache_20261015

Okay. Yes. Hello everyone. My name is Jin Young So, the senior director and system architect at Samsung Electronics. Today, I'm thrilled to share our latest advancement in CXL memory and SSD technology, specifically focusing on the agentic AI workload. Firstly, we will explore how prefix KV cache pooling and sharing are unlocking the new level of performance for agentic AI workload along with the CXL memory orchestration. Secondly, we will also talk about the SSD technology to handle the KV cache TCO effectively. As a large language model evolves, context lengths are growing exponentially, reaching up to 32 K and 128 K and beyond. This creates a massive bottleneck during the prefill phase. While the KV cache significantly accelerate the LLM inference, it notoriously consume massive amount of GPU memory HBM. The key idea behind the KV cache offloading is to migrate inactive or temporarily unused data from the GPU HBM to the low tiers like the CXL or SSD. If we can successfully keep multi-turn request data over shared prefix caches in memory, we can reduce inference latency by up to 25 times compared to the GPU recomputation. This offloading is no longer uh just an optimization, it is a necessity. Today, we will talk about the Samsung's two products for effectively offloading the KV cache. The one is the CXL memory and the other is the SSD storage. CXL memory complement main memory and HBM as a TCO effective solution for agentic AI KV cache through the native pooling and sharing technologies. SSD fundamentally provides a high-capacity storage that is impossible with traditional memory technologies and supports KV cache storage across various tiers through the optimization technology. Let's start from CXL. In agentic AI workload, the KV caches from previous requests must be maintained for succeeding ones. Uh but, because a KV cache size grows so rapidly while memory is limited, caches are often evicted earlier due to standard LRU policies, meaning that we could reissue the compute them all over again. Furthermore, requests using the same context often get fragmented across different servers, resulting in the cache miss. So, we need a shared KV cache pool to eliminate this unnecessary GPU re-computation. This brings us to Samsung's CXL memory module, CMM-D. Designed for the AI applications, this easily adoptable factor module utilizes a PCI Gen6 interface running CXL 3.2. It delivers a staggering capacity of to 5.5 TB and a bandwidth of 1.6 TB per second per system. With pin-to-pin idle latency at just 100 ns, the CMM-D is the good uh hardware for a massive and shared KV cache pool. To provide this at scale, we partnered with industry partners leaders like uh Marvell and Liquid to build a large-scale CXL appliance. Our initial proof of concept setup utilizes four CXL hosts and Blackwell GPUs connected to a CXL expansion system. Moving forward into 2027, our next-generation setup will scale up to 20 terabytes of memory capacity with a up to 5 terabytes per second bandwidth. This creates an enormous and centralized memory pool accessible across the fabric. Hardware alone is not enough. We need a robust software ecosystem. So, we have successfully integrated our CXL shared memory pool with the AdamCache and the VR RAM. By utilizing a custom CXL Dev Dex backend, the GPU workers can seamlessly retrieve and store KB cache chunks directly to the shared CXL memory pool. The storage manager orchestrate these operation behind the scenes, ensuring the GPU always has the token it needs without recontention. Let's look at a real-world agentic AI workload physical AI and robotics. Here, a vision language model access and higher-level planner taking continuous frame inputs from robot cameras to dictate actions like uh restocking groceries or wiping a table. This requires a long horizon multi-turn planning. The VRM uh must constantly reference past frames and actions putting continuous pressure on the KB cache. We benchmarked this multi-turn robotics workload on AI RAM architecture comparing our CXL shared memory pool against the traditional RDMA network approach. The results are highly compelling. When looking at the request throughput per turn, our CXL architecture achieved the 14 requests per second compared to the just seven for RDMA-based. Uh that is a two times increase in throughput simply by eliminating the network overhead and utilizing CXL the memory semantic shared memory pool. The latency improvements are even more dramatic. In the same multi-turn AGI bot workload, the CXL architecture delivered a TTF, time to first token, of 200 milliseconds. The RDMA setup lagged significantly at a 650 milliseconds. By sharing the KB cache over the CXL, we achieved over three times speedup in list parts, which is absolutely critical for real-time robotic operations. Finally, the we are bringing this to the cloud-native environment through the Samsung OCP Pangaea version 3 project. Agentic AI workloads are highly unpredictable in the view of the memory consumption. The memory demand can spike by 15 times during the two invocations. Through Kubernetes integration, our memory orchestrator can dynamically resize the pod memory capacity using standard DRAM interface. We can inject or remove the CXL memory dynamically while the AI serving pod continues running, ensuring resources are always exactly where the AI agent needs them. KB cache memory dynamics also need to be handled by the user-level software. Samsung OCP Pangaea version 3 also supports this type of memory dynamics, supporting the elastic devdex. That means the CXL memory is already for hyperscale level adoption for agentic AI and the physical AI service. Samsung is working together actively with the industry partner through the OCP workstream to make it clear. Yeah, this is all my presentation regarding the CXL. The next, my colleague Young-ha Choi will tell you about the SSD technology for agentic AI. >> Thanks, Jinny. Uh good afternoon. Uh my name >> is Young Choi and I'm with the NAND product planning team at Samsung. So, following the previous talks on CXL, I will introduce the next generation SSD technologies that Samsung is focusing on for KV cache applications. First, let's take a look at the KV cache offload infrastructure in common use today. Broadly speaking, there are four ways to offload KV cache to SSDs and storage. The first approach uses a mesh network that connects to local SSDs both inside each computer rack and between racks. The second is a memory expansion architecture with a dedicated storage rack. The third one is a high-performance file system server. For these first three, data transfer runs over the RDMA-based AI network within the AI infrastructure. And the last is block or object storage scaled out over the data center network of your TCP/IP. Most companies will choose among these systems depending on which type of a KV cache they mainly deal with. That raises the question, how do we categorize the KV cache that lives in these systems? That's what the next slide covers. As many of you KV As many of you know, KV cache falls into three classes. First, hot KV cache, which mainly serve active session KV decode prefill. Second, warm KV cache used for system prompts and multi-turn history. And third, cold KV cache, which holds things like documents and code repository. These classes are defined by a KV lifespan and the durability they require. The hotter the cache, the more memory-like it behaves, so it maps to HBM, local DRAM, or CXL. The colder it gets, the more storage-like it becomes, so it maps to block storage, object storage, or even HDDs. Warm sits somewhere in between. But what we believe that SSDs can themselves be really segmented along these tiers. Today, our main TLC SSDs, the PM1753 and PM1763, offer a balance of performance and capacity which makes them a good fit for warm heavy cash. That said, we are not just looking for to supply SSDs built on existing technology. We are also developing new technologies to push SSDs up to a higher tier than they can serve today. So, over the next few slides, I will show you the strategy of how we handle all tiers with the product and packing side of view. So, what you can see now in this slide is the world's smallest 256 TB SSD, PM1773. Uh incredibly compact with E3.S form factor. So, what's inside? It has a 392 TB QLC and then and a 5 nm controller. As for capacity, it ranges 16 to 256 TB and it is rated to 0.6 DWPD over 5 years. Compared to our previous generation, this product delivers twice the capacity and twice the endurance in terms of DWPD. As you may know, the level of capacity with appropriate endurance is a very good fit for long-lived source data and cold heavy cash. So, moving on, I'd like to talk you through the technologies we are developing for warm to hot heavy cash in detail in SSDs. The first up is a mixed media. The idea is to carve out on SLC partition inside a single TLC or QLC SSD, splitting the drive by purpose. In other words, one SSD handles two kinds of data, metadata and heavy cash data. Metadata is small but needs high endurance, so it goes into the SLC partition. Heavy cash data is large but doesn't need as much endurance, so it goes into the TLC or QLC partition. Both partitions are then exposed to the host. So, on the right, we are showing the DWPD gains from mixed media in arbitrary units. With today's TLC SSDs, the usual way to raise the DWPD is to trade off retention. Mixed media alone, without giving up any retention, can already raise DWPD substantially. And if we we are willing to sacrifice some retention as a trade-off, we can boost the DWPD even further. KV cache that needs high DWPD typically has a low retention requirements. So, we believe that used well, mixed media can enable multiple tiers within a single device. FDP is another way to tech that can raise DWPD. Using FDP with a large RUH, we can get larger DWPD by reducing to small WAF. This slide charts the relationship between retention and endurance I mentioned earlier and how it changes at different WAF levels. By combining the retention endurance trade-off with WAF control, we can tune drives from either the cold KV or warm to hot tier. Another approach is prefetching. This one takes advantage of how SSDs that store KV cache behave, especially because KV cache access is deterministic and the data can be pre fetched into DRAM. We can eliminate one hop. With direct transfers between the GPU and the PCI flash device, this one hop elimination improves both bandwidth and latency. As a result, we have seen TTFT in the previous stage drop by about 35% compared to a conventional SSD. The chart on the right shows those results. Here, hit refers to the TTFT when the full KV cache for the input tokens is reused. While this doesn't bring SSDs all the way up to the DRAM or HBM tier, you can see it close to the good part of the gap. So, today we have looked at how our Caching Strategy for the Agent AI era is coming together. From CXL memory readiness all the way to the SSDs and storage. First, on CXL memory, it is now natively supported and we are building out the hardware and software ecosystem together with our data center partners. Second, on SSD and storage, beyond the supplying a range of TLC and QLC based SSDs designed to fit Caching Cache offload infrastructure, we've advanced technologies such as FDP, mixed media, and pre-fetching and co-developing them with our partners. So, join us. We are always welcome to work with our partners. Thank you for your attention. >> Is that another Thank you.