video_OCP2026_AMD_Helios機櫃設計_20261015

So, good afternoon, everyone. It's good to be back at OCP. Uh today, we're going to talk uh in some more detail about our Helios uh rack-scale uh architecture. Um we've been sharing details over the past several months about um the customer traction, which is very exciting. Uh we've announced uh multi-gigawatt deals with uh three major customers uh with Anthropic, with Meta, and with OpenAI, uh as well as uh several cloud service providers who are scaling actively uh with us with this technology, Microsoft and Oracle, as well as Vultr and TensorWave and several others that are yet to be announced. So, today, um we've been what we thought would be really interesting is to kind of peel back the covers a little bit on the architecture. And uh really go into discussing some of the engineering decisions uh that went into uh the the overall architecture of the rack. Uh so, I I have uh I'm joined with on stage with by Steve Scott, uh corporate uh fellow uh and chief architect of our uh rack-scale system, as well as Alan Smith, who's our uh chief GPU architect and corporate fellow uh at AMD to to really discuss and and dig down into these engineering decisions. So, welcome. Let's get into it. >> All right. >> [clears throat] >> So, our first question is really just about what is unique when you go into a rack-scale design? Um as AI models grow in size and complexity and performance depends on how all the different portions of the rack interact together, um what changes when you design the rack as one accelerated system? And what does that enable for our customers? So, I'll ask you, Steve. >> Yeah, well, I mean, rack scale really permeates every part of the design. Um, when you think about it, the goal is to take a bunch of GPUs and very tightly couple them together. So, the first thing you have to do is have massive amounts of bandwidth, and you use a load stored direct architecture to allow them to to to communicate with very low efficiency. Um, and this is probably perhaps one of the last generations where we try to do this all electrically. So, first to start off, you have to have to design for very efficient electrical communication, board design, and and cable cartridge design, and you have to put as much close together as possible, which drives density. And so, the density then drives liquid cooling. You've got to get all that that power out, all that heat out, and so you can do that most efficiently with with liquid cooling. And so, that ripples all the way through the design. But then you have to worry about deployment scale. You're putting these things into data centers at massive scale, and so, uh, we designed it to have blind mate connections, where you just slide in a switch tray or compute tray, and all the power and the cooling and the communication gets made automatically. Um, you to to to reduce deployment friction, you have to have, uh, uh, turn key, uh, software to manage the thing at scale. You have to have resilience, et cetera. Um, co-design opportunities allow you to do things like packaging very high density NYX, and using, uh, our Infinity Fabric to connect to them, so you get higher bandwidth. Um, and so, you wrap this all together for the customers, and what it means is you get something that at rack scale works like a single GPU, uh, but has access to more memory, uh, and more network bandwidth, and more compute than you can get in any computing package. >> Thank Thanks a lot, Steve. So, that actually leads in really nicely to our second question, really about memory capacity and bandwidth because that's something that we've talked a lot about. It's been a a differentiator for AMD for several generations now. And so, um when we're running larger models with longer context, that really puts more demand on that memory subsystem. So, so Alan, how should customers think about the capacity needed to support the workload and the bandwidth really needed to sustain performance? Uh how do we think about that with Helios? >> Yeah, that's a good question. Yeah, like you said, you know, the the memory system of the GPU is something we've been innovating on for multiple generations on Instinct GPUs and and even before that with with the you know, original implementation of HBM GPUs uh for AMD. All right. Well, we're we're miked, so they may have to turn us up a little bit. Uh anyway, so um so yeah, so the memory bandwidth I think is easy to quantify. You know, we we all understand that memory bandwidth um directly impacts token generation or any other memory bandwidth bound operators, of which there are, you know, many in an AI workload. Uh the memory capacity is is maybe a little bit more difficult to um to directly, you know, quantify on on the the impact of workload performance, but the way that you want to think about it is that the the GPU has a certain amount of work that it needs to do in order to um you know, make good utilization of all the resources that are packed into the GPU, including the bandwidth of the memory, but the compute as well, um as well as the other pieces of the system that Steve mentioned in terms of scale up and and other things. So, the way that we think about it is um depending on the service level agreement that we have for the interactivity of the system, we want to be able to load as much of the model onto a GPU as possible. And you can think of that in terms of you know, how do I just parallelize the model? How do What's my parallelization strategy? We can leverage it through you know, traditional sharding and and putting different quantities of the weights on each GPU or that can be now through you know, things like mixture of experts and other ways other parallelization strategies that we have for the GPU. But we want to get you know, make use of the the computing capabilities of each GPU like I said. So we want to get as you know, few GPUs as you can in order to leverage the fact that the GPUs are generationally growing their capabilities per unit. And so that's one consideration and then additionally it's the users themselves. What's the amount of context that you're going to have you know, and the KB cash storage that you're going to need to go along with the models to support each of the users that are put onto the GPU. And ultimately depending on that that SLA on the interactivity of the GPU, we want to batch as many users as we can and increase the number of parallel users or concurrent users that we have on each of the GPUs. So as you look at all of those different strategies of effectively parallelizing the models and the users across the GPUs, the memory capacity becomes very you know, very important parameter that you can dial in based on the rest of the capabilities of the system. >> Yeah, keeping more of the data closer to the GPUs, less data movement and then of course the memory bandwidth being able to feed those engines as quickly as possible. >> That's right. >> Okay, Steve we'll go back to you. So we'll talk a little bit about scale up. So we've used we've chosen to use Ethernet for our scale up. So what drove that choice for Helios and what engineering choices enable us to meet both latency and bandwidth demands of these tightly coupled GPUs. >> Well, as we all know, Ethernet's been around for a long time. Uh gone through lots of evolutions. And um there are a lot of really attractive things about using Ethernet. Um the first is that there's this incredible ecosystem. You've got lots and lots of switch vendors and and signaling technology and connectors and and all of that. So, you've got you can leverage that ecosystem. Um you uh you also have management software, management control and telemetry and air handling all this. And so, we've got generations under our belt uh as do our many of our leading customers with with working with Ethernet. There's a certain familiarity. One of the other nice things about them is that Ethernet switches are on the forefront of switch rate X. Um and it turns out when you're trying to couple lots of things close together, um it's really beneficial to have high rate X switches. Um and as especially as we look more to the future and we're thinking about growing the size of the scale-up domains um even larger in future generations, having that high rate X switch architecture is is very helpful in terms of reducing the number of hops, not have to take multi-hop through a network, uh but be able to get things get to where you're going as efficiently as possible just by transiting a single switch. Now, there's a little bit of overhead associated with traditional Ethernet networks as well. We all know that the architecture. You've got these external NICs that have uh very complex um uh uh data movement engines and and the ability to address hundreds of thousands of different endpoints, etc. Uh with the amount of bandwidth that we had to put inside our scale-up pods, that's not a good design, right? And so, what we did is we took our Infinity Fabric or or UAL um uh protocol, which does direct loads and stores, and we built very very lightweight integra- tightly integrated NICs into the GPU fabric itself. And so, the GPU emits loads and stores directly packed into Ethernet packets. And it has a very sophisticated mechanism for doing this where um it will optimize either for latency or bandwidth dynamically depending upon the communication patterns. So, in areas where you are you have light traffic, but your latency sensitive, um it will very quickly emit packets as soon as the data is available. In situations where you've got high demand and you're more throughput bound, not latency bound, it'll pack more into the packets and become larger packets. So, it can very dynamically reconfigure between um low latency and and high bandwidth, uh but directly emitting loads and stores that just flow through a single switch. So, it ends up giving us the benefits of a very tightly coupled low latency efficient design, but then all of the ecosystem and software benefits um of leveraging the Ethernet world. >> Great. Um so, so Alan, I'll come back to you. So, when we think about the GPU, the scale-up fabric, all the software working together to improve workload performance, which customer metrics best demonstrate uh those benefits? >> Yeah. So, I think um there's a few a few metrics that we can think about in terms of what what how we demonstrate the the performance across the integrated system. Um I think it you know, at the high level, it's important as we scale performance of one that we have to make sure that we keep that scaling balanced across all the important parameters. So, you see, you know, as you look at the uh generational improvements of um the Helios system versus like the MI355 infrastructure that we had just a year ago, all the metrics have improved, right? So, we've scaled compute, we've scaled um you know, not just uh matrix compute, but also vector compute and other, you know, transcendental compute and other other types of compute, they all need to scale proportionally. Uh the memory capacity and bandwidth, which we already talked about, that needs to scale. The scale-up performance that Steve talked about, um and the number of GPUs within that scale-up domain, all of that scaled. And then also, um the cluster uh scale-out capabilities as well. Um and then of course, there's other architectural, you know, um goodies inside the GPUs and our our connectivity to the Epic CPUs and, you know, on and on and on. So, we we take a very holistic approach as we think about um scaling up all of that infrastructure together because the GPU and and all of that, you know, if you want to get the performance out of it, you've got to you've got to have everything kind of moving along with it. We don't want that to be um we don't want to lose utilization of the GPU, which is one of the most important uh from my point of view uh parameters inside the rack. So, we want to make sure everything else goes along with it. Um as far as the metrics that we look at, ultimately, all of these things need to deliver workload performance, right? And so, um you know, time to first token, tokens per second, tokens per second per users, which are all uh from an inferencing point of view, the the metrics that we are focused on. Um and moving that frontier Pareto, you know, up into the right, right? Every generation, we want to drive additional throughput, that's efficiencies, that's reducing the cost per token, but we also want to improve the ability to deliver more intelligent tokens um through interactivity. So, um that that's the metric that we focus on for inferencing. And then of course of course, for training, um slightly different things matter more um inside those parameters that I went over. Um but there, you know, we're trying to drive time to trained uh model of the specific, you know, capability. So, but again, like it's the workload performance that we want to focus on. >> So, we have different customers with different workloads. We spend a lot of time trying to think about, okay, how to drive performance per dollar and performance per watt ultimately on those workloads that they care about. Which takes us to our next question and and I'd really like both of you to comment on this if you could. We do a lot of co-design with our customers. We spend a lot of time trying to understand what they're solving for. So, how did co-design and our customers and partners influence Helios? And where do open interfaces give them flexibility? And then looking ahead, how are we working with them to optimize future systems for their workloads? >> I can start. I mean, we co-design is at the core of everything we do. I mean, we spend lots and lots of time talking with a lot of different customers, but you know, in particular, we spend most of our effort um talking to leading model developers, the the metas and open AIs and anthropics of the world as as well as the people that are hosting these large-scale data centers. You know, that some of them have their own data centers, but then you've got Microsoft and and Oracle etc. etc. So, we spend a lot of time working with them. It's really sort of two two-fold. One is we're talking to the to the workload folks to understand their workloads, where they are today, where they're going. And then and then we talk to the data center folks to understand integration into the data center and and reduce friction. Um and you'll notice that it's impacted the design in lots of different ways. So, the first thing you'll notice about Helios is it's wide, right? It's the OAR wide um standard. And why did we do that? We did that because in talking to people, um they they valued the extra width to give them flexibility. Um you've got more front panel space um to to have more network bandwidth, which is important. You've got more space inside in case, you know, for both what we're doing and what some of them want to do inside the in inside the design. It reduces floor loading, right? Um if you if you try to cram things too close together, the the floor loading becomes unmanageable. And so, all of these things factored into the physical design of of Helios. But, whereas we absolutely want to have reference designs and do have reference designs for for anyone who wants them, we also realize that one size doesn't fit fit all. And we've got lots of different data centers with different requirements. So, we built in flexibility everywhere to, you know, what are your what are your power feeds? What voltage are your power feeds? How what what flow rates and water temperatures do you have? Are you going for a no liquid cool no no chillers and so run it 40 45°C? Or do you have chillers and you're running at 25 or 30°C? Um, how much redundancy do you want? Do you want N plus N redundancy or or four to make three redundancy in the power system? Do you want super caps to ride through failures? So, lots of different flexibility sort of built in. The scale out networking is maybe one of the areas where there's the most flexibility in terms of do you want rail optimized multiplane, etc. And people have ideas for fitting it to their data centers. But, open standards in all this are are a must. We've always been we've worked very closely with OCP over the years. We we publicly released all of the the specs for Helios over a year ago. That helps you build robust supply system supply chains for connectors and power supplies and cooling and and everything. Even the things like rack movers. Um, and so, it allows people to to secure the right supply chains, to work with the ODM of their choice. And that flexibility has it really gives us the benefit to go into lots of different places. As we look to the future, I'd say probably two of the biggest things from a system level that we're thinking about are larger and larger domains scale up domains and and then maybe even the move towards optics. We're going to be sort of aggressively introducing optics in in the future. But, lots of stuff going on at the GPU level as well with the customers. >> Yeah. >> And so, it's fair to say everything from talking to the folks that are pouring concrete for data centers to the folks building the largest and most capable models in the planet today. >> 100% >> in between. >> Soup to nuts. >> Yeah. >> Yeah. And I will say, [clears throat] you know, just as important as the the hardware um interfaces that Steve talked about are the software interfaces. And one another example of co-design that I think is um has been very beneficial to the industry is the work that AMD's done with all all the partners that that Steve mentioned um through open compute on um numerics, right? So, the the work that we've done to um enable a standardized um microscaled um floating point formats and the ability to allow model builders to work through um a a common set of quantization and training that then works across multiple hardware vendors forming the you know, sort of the foundation of model portability, if you will, across those you know, multiple vendors. And I think that this is this is really critical because, you know, as Lisa talks about every time she speaks on the subject is that it's going to be a really large world of uh hardware out there running models for for inferencing. And um and given the ability to move those models with without having to go through requantization and and retraining um is really important. And I think that that's um that's you know, proving out as we see you know, more and more different accelerators and compute capabilities and even CPUs, you know, uh being used for inferencing. >> So, so those are all great points with you know, we we we did a lot of those things with Helios. When we look forward, how do we think about uh future co-design with our customers? More of the same? Are we going deeper in any specific areas? How do you think about that? >> Yeah. Maybe I'll I'll take one, right? think that it's becoming more and more important, right? Like the models are moving in extremely fast. Um it's important through internal work at AMD and through the the connections that we have with all the frontier labs and other you know, model builders and deployers um to really stay connected to what they're doing, uh to listen carefully about how they're thinking about the future. >> Talk frequently. >> Talk talk very frequently um so that we can all think about how um everything needs to evolve together because we are really in a an era where the models and the hardware are evolving together, right? We look at um how the models are changing and how we might want to respond with hardware uh adaption to, you know, give the ability for models to continue to scale. And then in the same time, the models are uh reacting to some of the things that we do in the hardware world as well. So it's just really important that we can, you know, talk very very frequently. >> Yeah. >> Holy cow. >> I I would say overall, it's just isn't exciting time. We we all in this in this room understand this. It's an exciting time uh to be working in this field because not only are are, you know, things moving fast for us, they're moving fast for our customers. And so the customers really have a hard time predicting. And so we're we are co-inventing the future together. >> Yeah. >> Yeah, I think that's a great way of stating it. So that's our time. So first of all, thank you, Steve. Thank you, Alan very much for for sharing your insight. Have a round of applause for Steve and Alan. Uh we encourage you to uh Lisa's giving uh the kicking off the keynote session in a couple of hours. So we encourage you to go and see her session and she'll be talking more about what other cool things we're working on for the next gen. Thank you. >> Thank you.