video_OCP2026_Microsoft_雲端規模電力與效率_20261015

Can you guys hear me all right? Awesome. Thanks everyone for joining us. We're having some technical difficulties, but I'll kind of get us started. >> [clears throat] >> First of all, congratulations for successfully checking in and making it here. I know it's been a little bit of a journey. Uh, so like I think we ran into some technical challenges, but we are getting set up shortly. Uh, So, I'm Brijesh Warrior. And my Here's Nitish, who's a colleague of mine. We are at We are from Microsoft. We're like engineering leaders at Microsoft. We basically focus on like we're part of the Azure compute org, and we basically focus on efficiency of our infrastructure. So, it's all about like how do we get the most throughput out of the infrastructure that we have deployed, be it like tokens, storage, compute, and all of that. >> [clears throat] >> And so, yeah. Oh, it's not going to work. Okay. So, the uh So, what Uh, so today what we're going to do is we'll talk about like the AI landscape. Like I think all of you here are very familiar with what's happening in the industry with the massive adoption of AI. So, we're going to talk about the kind of the implications of that, challenges as well as opportunities that presents for us. And then we'll kind of talk about like how we are addressing this like how we're going to get working on like improving efficiency of that infrastructure. So, we're going to talk kind of go behind the scenes on like what we're doing at Microsoft, like sharing some of the best practices, and like we'll share a few examples of how we are actually kind of like specific examples of how we are doing this, and including like a set of papers that we have published published on this topic. Uh, so I think last year at this forum like I think Okay. So, I think we are up and running. Uh It's cool. You can be relieved. So uh so I think like I said, so I think last year at this forum uh we talked like one dimension of the AI challenge, which was around like uh power stabilization. So I think all the massive power swings that happens when we do AI training. So we had shared details on that. Since then we have made significant progress and we have some talks in like some additional talks here later where uh we talk about some contributions we are making to OCP around like uh capacitor-based backup units in the rack and so on. So today we are going to kind of step a little back and talk more broadly about how we are managing uh software like managing power through software. So first to kind of motivate the conversation, like I'll kind of talk about like the growing AI demand and how it's affecting like this demand on like a few dimensions. So one is like power. Like so I think if you look at like the IEA projections, by 2035 we expect to be at like 2 and 1/2 X in terms of the size of the power deployments. And this is at the conservative end. Could go all the way to like 3 X by like 2035. Another dimension of the scale is around like servers. So I think year over year in 2026 we're expecting like 31% more AI servers being deployed. And like the broader AI server I mean broader server deployment is increasing by 12.88%. I think one interesting thing that we are also noticing is that uh as there is this massive adoption of AI scenarios, there's a lot of this agentic AI scenarios that has is also putting pressure on real like the traditional compute. So there is a demand across like GPU, storage, and compute. Uh and I think like the third dimension is like CAPEX. To fund all this expansion, there is like a massive CAPEX. Just across >> [clears throat] >> the big hyperscalers, we are seeing like $650 billion of CAPEX being spent in like 2026. If you include the new clouds, it's like in the trillions of dollars. So I I what this is highlighting is that there is like a significant like an accelerated demand for AI and there is like there is a limits on how much we can build. So, I think it's super important that we get the most out of what we are building. And like so the basically the focus needs to be on efficiency. So, I think that's kind of the motivation for our conversation today. So, Nitish is going to join me here to talk about how we are leveraging like software to kind of think about this problem holistically and drive efficiency. So, I think the opportunity here is massive. Like I just said like if just look at the $650 billion spent. Even if we have like a 1% improvement in efficiency, that's like a $6.5 billion in savings. And that we believe we can do a lot more than that. So, with that I'll hand it over to Nitish who's going to walk us through the like different initiatives that Microsoft is driving. >> Cool. Thank you, Brijesh. So, meeting demand with less burn, right? So, for us the largest lever that we have is power. If we can make sure that we use the power that we have efficiently, then we would be able to fit in more servers into our existing data center space. But power is a really hard problem. It's It's It's a cross-stack problem that we need to solve. If you think about it, compute, networking, and storage, they are all software defined. We pull them, we schedule them. We as an industry solved this 15 years back. But power for most part of it is still very static. The decision about power in a data center happens in three places predominantly. The first is the infrastructure, which is where the limits of what all the IT capacity can consume is set. The second is the software, which basically decides where the work that needs to be run lands. And the third is the hardware that decides how this work runs. Each of this are traditionally tuned on its own. So, what I mean by that is the facility team when we provision the hardware, they provision for the nameplate. Though it feels very safe to do that, it leaves a lot of power stranded because not every rack, every server consumes its total power and that it's expected to use. The scheduler, the software, tries to pack work very tightly. Really works well for efficiency, but leaves a lot of tight limits that we hit as we look at the hierarchy of the system. The third is basically the hardware capping, which is basically the backstop, think of it, but it sounds safe to do a backstop system, but then you impact both critical and non-critical workload the same. Now, each of this looks right on its own, but together these are hurting us. So, power has to become software-defined. Now, this is a problem that we have been working on in Azure for almost a decade right now. So, in this talk, I'm going to talk about what we have done here and how we are adopting some of these learnings to this AI age. So, what is software-defined power? It basically starts with three inputs at a very high level. The electrical topology that you have, which is where the racks and the the IT infrastructure gets uh deployed. The power telemetry, which really tells us of how much power is actually used or drawn in the data center. And the third is the workload, which is the service context. What workloads can sustain what amount of throttling, what flexibility do they provide, and all of those details. These three inputs feed into a very simple loop where we observe these three inputs. We decide what is the most optimal decision that we can make make in order to satisfy all of these. And then we act on them. And of course, there's a feedback loop that goes back and says like are we doing Did we make the right decision? On the action, we basically have like two levels of like actions that we like do. One what we call as a power steering. And the idea here is you basically take the power to the work. And the second is power shaping where you take the work to where the power is. And we'll talk about this a little bit more in detail in the future slide. So, underneath all of this is the hardware backstop. So, this is the protection I mentioned, right? So, while we are doing all of this, think of hardware as as the safety net that we always have in the system. So, while software is trying to make things run efficiently, the hardware helps us keep it safe. More importantly, this is not a single loop. There are many such loops that run through different levels and work at different speed. And at the end when we conclude, we'll see how of how this makes sense. So, let's start with steering. So, if you look at a data center uh power system, you can think of it as a hierarchy typically. And usually in this hierarchy, you have budgets. With steering, the idea is what if each of these levels could manage its own budget? For example, at the top here, you have the electrical domain which is usually the cell of the rows. Now, these have power budgets. And while these rows have the power budget, they can distribute this budget to the racks that they host. Similarly, the racks host servers and they can distribute the budget they have to the servers. The servers have workloads. The workloads have priorities. They can distribute the budget they have. And ultimately, you have the hardware. So, the telemetry is typically like flowing up this system. While the budget or or the caps are are flowing down. So, each at each of this layer, as I described, the budget is steered across these different levels. Now, this fundamentally changes how we build for power. So, no more we need to build for the worst case. We plan for the real demand. And when we group the racks to use this power budget that's available, the rest of the headroom that is available, the slack, what we call as the slack, can be lent to the flexible internal workloads that could shed when we need to. Now, let's look at like how we are adopting this to AI inference. So, as you all know, inference a inference request is is really of two types of different jobs. So, there is a phase called the prefill, which basically reads the whole prompt at once, really compute heavy, very power hungry. And there is the second phase, which is the decode, which starts to generate the the response to that request one token at a time, very memory bound, but not power hungry. Now, you can think of like the prefill here as setting the peaks, while the decode is leaving the headroom that's available. So, when we can give each of these phases the hardware that it needs, now you can steer power based on the workload demand. So, this came off our research from Splitwise. Um we're very happy to see the industry adopting this today. And it's completely changed the way that we provision for power for inference in Azure. Uh there are few QR codes that I've like put on some of the slides here, which are all publications that we have done as part of this work. Uh feel free to like scan them, and if you're hungry to get into the details, you'll find more technical details in those um papers. So, now let's go to shaping. So, we talked so far about moving power to the work. Now, shaping is about moving power to moving the work where the power is. If you look at schedulers, traditional schedulers, they look at all servers as the same. Right? But, in reality, when when you deploy these servers, when you deploy these racks, they're deployed across power domains. So, in this example here, I've shown two power domains, A and B. And uh the idea here is each power domain is near its limit, and one of the power domains has a lot of headroom. But, the scheduler does not have this information. With power shaping, the idea is if we could expose this information to the scheduler, the scheduler could now be smart enough on where to schedule this work. So, in this animation here, we showed like how the work could be scheduled on a domain where there was sufficient amount of power while not risking it and putting it on a domain that is power hungry. Now, this is a real power room that we're talking about. Sorry, for some reason the animation did not like show up there. Uh but, it it was supposed to be a graph that basically shows that in Azure, uh you will find that in the paper in the last link that you had, where if you look at all the different rows uh in in in the data center, we see not all rows running hotter. Like when few of the rows are running hotter, there are few rows that are running really cold. And that lets us balance the power across these rows. That's So, now taking this to inference, right? So, we do this shaping in the scheduler, we are adopting this for our GPU scheduling, but we have opportunity to even push this to inference. So, inference request, think of it a very simple way. You have an inference request that comes in, you redirect that request to an endpoint, right? You have multiple such endpoints, these endpoints are distributed across power domains because of fault tolerance that you need to like maintain. If we could provide this power information to to this routing layer, then this inference request that comes could now be routed smartly to the endpoint which could handle it without hitting the power limits. So, let's remember that this is this is an optimization. Of course, the latency and quality targets are still met, but providing this power information makes this whole stack smart about balancing the power. So, you don't again have to like provision or design for the worst case. So, this other the paper that we have like published, um feel free to scan it if you need more details. So, we talked about two instances here, like taking work to where the power is and taking power to where the work is, but what if there's no real power now? So, now we talk about like power capping, right? So, when we started looking at this, I mean, these are rare, but these do happen, and you need to manage them smartly. So, we wanted to look into what are the opportunities when we have to cap power. This is during real emergencies that you need to act. We were thinking of what smartness can we get into into into the capping infrastructure. So, the services can tell us three things. They can tell us the priorities, they can tell us how flexible they are, we could measure this, and what are the performance constraints they have in the system. So, if we can provide this information to the policy to to to the thing, we could be smart about categorizing what work that's being running in the data center is critical and which is not. And this can really help us in in shaving power when we need to shave. But why do you think the the capping flexible is great? Again, animation fail. Sorry about that. But again, in the paper that I showed in the last uh slide, uh if you go through it, we have actually shown that in real inference model workload, where we are uh handling the inference request, even a little bit of throttling, a little bit of uh power throttling could shave a lot of power. So, it's not really bad during an emergency, but of course, we try to use power steering and power shaping here to make sure that we don't hit. But we have gone beyond. We have taken this one step beyond. And we have integrated this whole solution into a hardware. Cobalt is our home-grown uh CPU system. And as I mentioned to you, capping typically happens at a hardware. It really doesn't know or have any intelligence into what gets capped. So, we built on this using software, where we could categorize what workload is important, critical, non-critical, and we could we could throttle them. But that was still slower. It would still take multiple seconds for us to engage, disengage, talk to the hardware, use the DVFS or or use the uh frequency capping, whatever the levers the hardware has. Now, what if the hardware could take care of this for us? And that's exactly what we did with Cobalt. We integrated all of this algorithm into into our hardware and we enrich our operating system to provide insights into what workload is running on the system. And using this information the hardware now can throttle, manage the workloads much better than what we could do with software. Uh we published an article about a technical article about this work recently early this year and the data shows us the Cobalt 200 which is what uh we have deployed running in Azure we could over subscribe up to 20% more than what we could do traditionally without impacting any critical workload. So, putting it all together here. So, these are all the different works that happens at different layer. And I talked to you about in the first slide there's not one control loop, there's many control loops. So, we saw control loops that happen in milliseconds to seconds range. These are the capping system that is uh this is the shaping system that tries to route the request for inference. This is the steering system that tries to give budget to different types of workloads, the prefill phase, the decode phase. Now, all of this needs to operate in a milliseconds scale. And then you have systems which can do placement, the scheduling systems which could be a little slower. These are the placement systems that we have that we have enriched. And then you have the months and the year time frame so these are the planning systems that we have. So, this is the Uber thing that you do with planning of heterogeneous hardware, how do you plan for a grid power and all of that. So, if you see this and how we see this in Azure is the fast loop that we have here that's for our protection. The slow loops that you see here are for optimization. And software defined power is what makes them all work together. So, that's software defined power more AI for every what we already have. So, why are we here in OCP? So, today I and Brijesh we showed you like what we do in Azure and and how we are managing power which is a really, really constrained resource today. But, as we build this, as we work with every new vendor, as we build with every new hardware, we need to wire up power telemetry. We have to run through new control systems. And this is something that doesn't scale for us. And I'm pretty sure it doesn't scale for anyone. So, we are here at OCP and we see three opportunities that we want to ask the community to help with. One, the open telemetry. Standardize the power and the capping signals. That's just an example from this software defined power that I just talked about. Standardize the way those get reported. Open topology. You build the power infrastructure, the the data centers. How can you like expose that in a programmatic way that so that the control systems, the control plane systems can understand and and work with it? And the open controls which is the hardware controls that that you have or you engage to control all the different power policies. So, OCP groups like the uh the hardware management group, uh the the rack and the power are natural places for this to work. And we would love to build this together with you all. With that, thank you.