video_OCP2026_NVIDIA_GW級AI工廠網路_20261015

I'm I know I'm going to pay for it later on, so. Uh so, let's talk about what what NVIDIA builds or what what NVIDIA design. So, we build or design a full AI factory. And essentially, it's a single unit of computing. It's a full stack single unit of computing that is built for generative AI. Now, there is no when you build an AI factory, it's not just putting one network. Actually, there is five networking infrastructure within that single unit of computing. We start with scale up. And the mission of scale up is essentially to build a GPU unit. So, there is massive amount of bandwidth, extremely latency, in-network computing, connecting GPUs to actually form a single unit. Um because of the massive amount of bandwidth, we're trying to use copper as much as we can. And we're trying to increase the compute capacity within a rack. So, we introduce liquid cooling and actually hot liquid cooling, bringing 800 V to the racks, be able to extend or increase the capacity of the of compute so we can extend the usage of copper. After you we build the single unit of GPU, a rack-scale GPU, we need to scale it out to hundreds of thousands of GPUs, and for that we have a scale-out infrastructure, Spectrum X Ethernet or InfiniBand. There are two options for that. Um that infrastructure is all based on optics. Because of the distance, it's based on optics. And we are building or brought a a great CPO infrastructure to cover optics and minimize power consumption and increase resiliency, and And touch a little bit on that later on. Sometimes the compute capacity in one location is not enough and we need to extend that. And the only way to do that is with scale across, which is ability to connect remote data centers, but create that infrastructure to run a single workload on top of them. Obviously, we have latency that we need to hide. And we're trying not to increase the penalty of that and therefore we build an infrastructure that does not rely on deep buffers that may create extra penalty of latency. For Gent K I, we also created a new infrastructure for storage, which is context scale, completely optimized for KV cache, completely optimized for context. We're using Blue Field as a storage processor, so we can run the entire storage stack for KV cache, for context on it. And with that be able to actually increase the token rate and reduce power the power consumption of our storage infrastructure. Recently, we announced a new one. It's a new infrastructure that replaces the traditional front end or traditional north-south or traditional access network. And we call it scale in. And the reason that we call it scale in is because essentially it's being used to scale in models and users and data and storage into the air factory. But it also need to make sure that there is security all the way. So, traditional front end network was just dealing with front end. But in air factory, that infrastructure actually connects to any all other infrastructures. Because it needs to ensure security for accessing, it needs to ensure security running within, it needs to ensure security for agents, for example. And therefore that infrastructure connects to scale up, to scale out, to scale across, and to context scale. It's an infrastructure that maintain and govern security across the entire other four infrastructures. So, for scale out, we built Spectrum-X Ethernet, and we created an infrastructure specifically for distributed computing workloads. Brought lossless because we don't want to create jitter of dropping packets because of congestion. Focus on RDMA. Bring telemetry-based congestion control and doing sophisticated adaptive routing, which resulted in an infrastructure that enable single workloads to run on multiple hundreds of thousands of GPUs. Um higher effective bandwidth, greatly supporting multiple agents, multiple users running on the same infrastructure. Now, Spectrum-X Ethernet is being used in many, many, many systems, um and has support for variety of operating systems. FBOSS for Meta, SONiC, um Cumulus is a reporting system that we built, and Nexus OS. Great partnership with Cisco. Cisco is leveraging our Spectrum-X Ethernet ASICs to bring the AI performance into their infrastructures, and they're running their operating system on top of that. We did announce this week uh first 100 terabit switch for Meta, um Spectrum-6. Um and it's not just a 100 terabit switch. We work with Meta on building a multiplane topology. And not just a multiplane topology to reduce the tiers of switches and enable larger scale, but also utilizing the hardware load balancing that we enabled. Hardware load balancing that we built in order to make sure that we can real-time react to changes and real-time react to failures. So, if one plane is down, we can quickly load balancing on the remaining plane and so forth. So, we built a hardware-based multi-plane infrastructure with Spectrum-X together with Meta, so they can use it on prem and they can use it in the cloud. And there is lot of uh more work that we are collaborating together in order to actually increase that scale even further. So, why uh why Spectrum-X or what we did there? The ability of Spectrum-X to focus the switches on unconditional adaptive routing and leveraging the super NICs to place data in back in order on the GP memory directly or CPU memory directly. The ability of the NICs to collect telemetry from the switches and be able to control injection rates based on what's happening on the on the network enable us to actually provide consistent performance. Consistent performance for AI workloads, for distributed computing workloads, and it's become even more evident when we're running multiple workloads on the same infrastructure. So, in traditional Ethernet, once you run multiple workloads on the same switch, one workload impact another. With Spectrum-X, there is no impact. With adaptive routing, hardware-based adaptive routing congestion control, we can actually run multiple workloads on the same switch and one doesn't impact the other. And we're getting almost 2x of performance improvement in those cases. Another thing that we announce uh this week is uh a new optical module, USFP. And the reason for USFP is obviously to enable the next level of density. The amount of bandwidth per compute devices doubling from generation to generation easily. Switches increases in redix. We need higher density module module optical modules, not just for the switches, but also for the compute nodes. Today every GPU has a 1.6 teranic attached to it. Next generation is going to go double, for example. So there's a good amount of bandwidth that just come from the compute side. So we wanted to build something that has provide the right density. We also want to build something that is simple to deploy. It doesn't have a different or separate liquid cooling circuits and complexity of that sort. We also wanted to build something that can support 400 gig. So this is the the benefits or the essence or the elements behind USFP. Um we're doing that in collaboration with Cisco, TE, Amphenol, um and that is our optical module for pluggable devices both on switches and on um compute infrastructure. So going to co-package optics, going to co-package optics, the drive is simple. We started with scale out because scale out is completely based on optics. And today when we look on the on the amount of bandwidth or the amount of network that we use for scale out, that optical network consume power that is equivalent to almost 10% of compute. So obviously if we can reduce that, we can bring more compute and we can generate more tokens and our customers can make more money. So the reason is obvious. Um and we started with scale out. Now, later on we also going to move to scale up. And we mentioned as the scale up right uh size continue to increase, within a rack we still want to use copper cuz there is nothing better than copper but once you go rack to rack we are going to use copper cage optics as well. So today after good amount of work and creating innovations in multiple aspects with a lot of partners of ours we build optical engines that is based on micro ring modulator. We designed the the fiber the way that we attach the fiber array in a different way. The packaging process with TSMC coop in order to make sure that we bring all the resiliency into the building process itself. Today photonics and video photonics is in production. So those are images from the different areas. TSMC this is the the coop process actually testing placing the electronic IC with the photonic IC the lasers and so forth. Um and we're shipping. So in in combination of CPO on one side and what we do on multiplane on the other side actually bring or build a great scale out infrastructure. With copper cage optics we can reduce power consumption up to 5x. And MTBI is increasing by 10x as well. I think in the meta presentation the first one actually also demonstrated a 10x improvement in MTBI and this consistent with what we see running hundreds of millions of hours on on copper cage optics. Um with the multiplane topology we also bringing more resiliency by itself. So it's a great combination of CPO and multiplane. And this is example of what we build in the in our multiplane network design. When you have many ports or many planes that are coming from the NIC, you need to do load balancing between them. And the simple way to do load balancing is do that on the software side. And doing load balancing on the software side from multi-port NICs it's not something new. The problem with that is that in a case of a failure, you need to do First, you need to detect it. And then you need to do reload balancing on the rest of the planes. And that takes time. We implemented that load balancing in hardware. And we brought the same IP that we placed on the switch on Spectrum-X Ethernet for doing that adaptive routing. We brought that all the way to the SuperNIC. So, now the SuperNIC can detect it. And it will do load balancing based on the switches that it connected to and based on their conditions as well. So, instead of waiting almost 1 second to detect a failure in multi-plane and then to react to that, we can do that in less than 3 milliseconds. That's a huge difference. So, there's a quick detection and then completely re- reload balancing on the rain remaining of the planes. So, if one plane is down, if a switch is down, we still have 90% of bandwidth. Now, when the link is back up, you want to get 100% bandwidth as soon as you can. And again, that detection in hardware enable us to do that very, very quickly. And move back to 100% while in a software approach it takes much longer time. So, the benefit of multi-plane enables to build larger infrastructure with two level of of of factory networks in a sense. And the combination of that with co-packaged optics bring the power reduction, increase longer time of MTBI, higher resiliency, and ensuring that everything is keeping running even in case of failures. The The last part I wanted to mention is scale across. Um and in scale across, we wanted to enable harvesting the compute power of remote data centers to run a single workload. The traditional way to connect um long distances is based on deep buffers. Because essentially you don't know what's going to happen on the path, and it's better to have a good chunk of servers in case of congestion. The problem with um deep buffers is that they create latency penalty, that it could be 2x and 3x the penalty of the distance itself. So, we need to pay 5 ns a meter, but I don't want to pay two times or three times of that latency. And therefore, we wanted to develop something that does not use deep buffers. And the way to do that is essentially be able to measure the measure the distance. So, when the world starts, we actually send telemetry probes across the paths that exist in order to measure distances, in order to measure latencies, in essence. And every NIC, every GPU might have different connections to different places, which each connection has a different latency or different distance. And by understanding the distance, we can modify the algorithms the the data distribution algorithm based on distance. And by doing that, we can actually connect everything on long distance without the use for deep buffers. The result that we're seeing is almost seeing almost double performance of nickel using Spectrum X on on the long distance. Um so, that's something that is already in deployment today. We're connecting tens of kilometers between data centers, and by the end of the year, we're going to go and start connecting hundreds of data centers across hundreds of kilometers apart. So, my last slide cuz I got the 1-minute extra. Uh five infrastructures, it's not one, it's five different infrastructures. We have uh scale-in and scale-in today, by the way, run the security for the entire infrastructures. Not Not just access. It's controlled the security on scale-out. It's controlled the security on scale-across. It controls security on storage. It monitor security for agents using in-silicon out-of-band telemetry leveraging uh BlueField 4 as a DPU. Um scale-up and V-Link running um as much as we can within a rack using copper. Once it's going to go outside, we're going to leverage co-packaged optics. Um scale-out is now doing transition uh to CPO. Then we have the storage uh and scale-across. And with that, I want to thank you very much for listening. >> All right. Thank you, Gilad. >> [clears throat] >> So, formally, this is a coffee break time, but [clears throat] informally, we're going to keep Gilad on stage and talk through half of the break. Um So, I'll take a couple of questions from the audience, but I'll start with a question from me. >> Not the previous question, but okay. Yeah. Yeah. >> I'll rephrase it. I'll rephrase it. So, CPO is shipping. It took Nvidia 7 years to develop it to get it out. >> years? >> Seven, I think. >> Seven years? >> Yes. >> No, I think it's like probably 2 years, but yeah. >> Well, you're ruining You're ruining my question. >> Who's counting? Who's counting? >> my question. I'm counting. We like counting. We're counting. It took 7 years for Nvidia working with TSMC to develop this technology. Some customers are smart enough to deploy it now. A lot of others are kicking tires and saying, "Oh, we can do it better. We can develop something ourselves a lot faster. It will work better." Are they fooling themselves? Because we know CPO will work better. Why do they bother going with other variants of it? >> Okay, I think I think there is usage for a lot of different technologies in different places. You know, there's obvious questions if you have CPO, why are you why are we doing USFP? Why aren't we investing in pluggables? It's because pluggables has a place. CPO is built for specific ranges, for example, but you use transceivers for longer distances, use transceiver for other applications, and so forth. When you go CPO, you choose an optical engine, right? You choose a specific technology, and we choose the one that makes sense to cover what we need within an air factory. You want to go outside, you need pluggables. You want to connect to a a different infrastructure, may need pluggables, and so forth. And therefore, there is a place for both. And that's why we're building both, and that's why we're also investing in pluggables. And USFP, it's a great simple design, very dense, supporting 400 gigs. So, we have the path forward the future. And with the um amount of lanes that we have there, we can do easily 6.4 ter and then always also 12.8 ter on pluggables. Um so, it's a it's a it's a great um it's a great new generation of pluggable models in a sense. Um you heard they ask about NPO and CPO. Sometimes you will need to do NPO. You know, we're building a CPO infrastructure, it's a great technology, performance is great. Uh bit error rate without FEC, it's better than any pluggable today even even see. So, and we know that once you go NPO, you're going to lose some performance cuz you're going to be lost that you need to handle. So, if you can do CPO and new CPO, it's a great technology. Now, you may have other designs that will be too dense, so it's going to be hard to use that from a big point perspective and you may want to use NPO in the future as well. It's not good or bad. I think there's probably a place for all of those. I think for scale out and looking on scale up, CPO is great. It's actually we build what we what we need and it's provide great performance. And we start shipping, so you start hearing more and more feedback about that and we see great results. Um if we need to do NPO, we'll do NPO for other places. You know, we have the elements, we have the technology and obviously pluggables. >> Right. Right. So, with the USFP, it the photos that you shared, it looks very similar to OSFP in terms of size, but it >> little bit larger. It's a little bit smaller, actually. >> It's smaller than OSFP. >> smaller. I can show you one if you want to see. >> What is maximum power? What is the maximum power dissipation? >> Uh it can support up to 100 watts. >> 100 watts. So, you will probably have to use LROs for to meet the budget. >> Uh not necessarily. I think you can have support because it support up to that, you can actually support different kind of connections. Um and it's from a dimension perspective, it's a little bit smaller than OSFP. >> Wow. >> And it brings um you know, six on the 200 gig it brings 6.4 or you can go 6.4 terabytes the 1.6 that you see today. >> Wow. Wow. When do you when do you think we'll see first deployments of USFP? >> Uh I think that you're going to see good demonstrations not long from now um and then we'll go to production, so it's going to be next year. >> Next year. Wow. That's fast. Any questions from the audience? No questions. Everything was perfectly clear. Well, thank you very much, Kevin. And let us give another round of applause. Thank you.