video_OCP2026_Meta_主題演講_GW級AI基礎設施_20261015

Hey folks. I was just talking to Lisa backstage. It's never a great idea to follow sort of the Rolling Stones on stage, right? And I guess that's what's happening to me. Uh Hey folks, my name is Santosh Santosh Janardan. I head up infrastructure at Meta and engineering. Which is a fancy way to say that if your Instagram, your WhatsApp, your Facebook's not working, I'm the one to blame. Okay. And people genuinely do that. You should see my WhatsApp messages. Uh listen, the these days we can't seem to construct a sentence without the word AI in it, right? Uh about 2 years ago I want to say in '24, we were just finishing up creating the first class our first really big cluster, right? And it was decent size, over 100,000 H100s, uh almost 150 MW. And uh the results were good. Uh the models were coming out fine, and the scaling laws were holding up fine. Now, all of you knows what what does that mean? Well, that means that the next cluster has to be an order of magnitude bigger. Right? Now, see, I work at Meta. We are used to scale. We have mul- multiple gigawatts, tens of millions of servers, and we do a fairly good job of serving what, 3 to 4 billion people on the planet. But AI infrastructure is very different. Different constraints, different sensibilities, different scale. So, how we went about sort of creating a next new culture was quite an epic journey, right? So, that's what I want to hopefully walk you through. And the cluster we built is in Ohio, in any of New Albany, Ohio. And we call it Prometheus. Now, Prometheus is probably the single most ambitious at least infrastructure product that we have ever attempted. And remember, we are actually also training a model while we were building it. In fact, today we release our latest model. It's It's a frontier model called Muse uh uh and there's there's a bunch of uh sort of usages that you've seen for the models. All of them were trained right here. Now, this is pretty big, over 30 structures, millions of miles of fiber cables, tens of thousands of GPUs. Now, construct constructing it together, like I was saying, ended up being quite an epic journey. So, what I want to do today is actually take you behind the scenes. Give you a glimpse of what it actually takes to build something that is of this magnitude. Right? And when we do this, whenever you do a huge project, you do a bunch of assumptions. And the assumptions are pretty simple. When they hold up, life is simple. When they don't, you go and tackle them. Some of our assumptions didn't hold up. The first one was that a cluster lives in one single roof. Now, it makes intuitive sense if you think about it, right? But, the way we were thinking about Prometheus is the fact that there were over 30 different structures. They're all of different uh shapes and sizes, spread across sort of different proximity distances. And what you quickly end up realizing when you see this is that network is a huge problem. Why? Why is network such such a big problem? Well, it turns out that when we when you do AI training, AI training moves forward at the speed of the slowest conversation that you can have between two GPUs. Plain and simple. Now, when you have all kinds of sort of distances sprinkled in, it becomes harder. Right? Physics is simple, folks. Distance equals latency. It takes about 5 microseconds for a piece of data to go at the speed of light for about a kilometer. So, let's say it's 8 microseconds if you want to do a mile. So, if you do a round trip, it's 16 microseconds. Simple math. Now, if you do millions of round trips per sort of training call, you quickly add up the time, and that's where sort of the losses end up happening. Now, in addition in AI training, there's another constraint. You don't want to drop packets. You essentially want to create a lossless network. That's not how the internet works, right? So, now if you think about those two constraints, you have to go and solve it before you can make forward progress on anything. So, what did we do? Well, we went and created our own sort of backend aggregation network or the bag. Now, in essence, this is an Ethernet super spine. You take the spines of all the buildings in a region, string them together, aggregate them locally, and when you go over longer distances, you go over the you go back to back, so to speak, right? Now, this gives us two advantages. The first one is that you get deep buffers at the bag layer. So, this gives us a little bit of uh headroom so that you can deal with the lossless thing that I was referring to earlier. Second, this is distributed. There are no SPOFs. No single points of failure, and that really helps because it is resilient by design. Now, the other thing to understand here is that this thing is fast. Tens of petabits a second. You can transmit the whole Library of Congress in less than 30 seconds flat. This thing really flies. Now, when you do training, you have to have GPUs talk to a GPU, right? That software that does that is basically called the transport, and industry standard, like all of you you is Rocky. Now, when you use rocky, you send bits down a software spine. It goes, sometimes hits a hiccup, and as as the diagram shows, it hits a red block. It backs up. So, we created our own thing called meta rocky. And what meta rocky does is it takes the same bits, sprays it across multiple paths, and actively monitors the transmission of every single bit going through the path. Every single time it hits a block, it routes around it. And life continues as normal, and it retransmits only that particular bit, not everything else. Now, this has been a huge improvement for us in terms of throughput, in terms of reliability, in terms of resiliency. And we uh are actually open sourcing the software, the test harness, the test suite, all of this to OCP. So, all of you are welcome to use it. We'll learn and grow together. Networks are physical as well. All the networking thing I was talking about, this is the actual physical layout between the buildings. Each one of those pipes that you see are serving actually thousands of strands of fiber. Literally thousands. In fact, if you aggregate all the network fiber we have in Prometheus, it's about 3 to 4 million kilometers of fiber. You can go to the moon and back four times. Physical sort of manifestations of where I'm draining that truly boggles your mind. The other thing to call out here is that I've been talking about sort of the data center, the physical layout. The same thing exists at the rack level. If you think about it, every every sort of blade in a rack also needs bandwidth. And the more uh you put into it, at the end of the day, you you hit physical limits. You just can't fit enough copper uh in it to ac- actually do what you want to do. So, what did we do? We created a new optical backplane. Uh we we actually have a prototype of this in the in the meta booth. So, please go check it out. But, fundamentally what this does is that it creates optical backplane, same rack, same serviceability, same cooling. So, nothing else changes. So, hopefully we can just implement this and just take an order of magnitude step up in terms of bandwidth. Second assumption we had was that data centers are built from concrete and steel. Again, makes intuitive sense. You want some You want something that you know, sort of sustains over time, weather, etc. Now, but it takes time. Conventionally, when somebody comes and asks me that can I have a data center, my usual response is, "Well, you should have talked to me 2 years ago." It takes time. So, because but time to market matters. So, what we ended up doing is that we ended up going and re- started rethinking exactly what exactly goes into a data center in the first place and started questioning ourselves. Can we just put it under a tent? Now, this is a point usually in a presentation that I ask the audience to imagine a big tent. Look around you. This is an example, in fact, it's the same company, Sprung Structures, right? That has created a tent. Now, the difference here is we went really big. Each one of our tents is the length of minimum two football fields. This is a humongous, folks. This goes on and on and on. And the one thing to understand here is that you take a big concrete pad, lay it out, and and you create these things. This takes months to build, not years. This is built to the same standards of a data center. It is rated against the category one hurricane, so it's fairly robust. Because you put in billions of dollars worth of equipment in it. But this is what you need when you go want to go uh really fast. But it's not just uh building the structures. We started rethinking what goes into it. It's not just building a shell. What is the minimum viable sort of uh data center sort of scaffolding you need to put the servers in it. That was the real win in my mind. It's not just that you build it a tent. We got the minimum cooling. We got the minimum power, not more, not less. And overall, it's it this goes uh online in between 3 to 5 months and it's way cheaper. So, win-win in multiple situations, right? Here's the fun part. You build a tent. You have to put the servers somewhere as well. I just can't seem to go back, but if somebody can play this, awesome. Okay. This was supposed to be a fancy video showing you how concrete is poured. Hopefully, it'll work, okay? The The way to think about this is concrete is something that I don't I don't think people understand how much concrete is used in the world. Concrete is the second most used material on the planet. Second only to water. That's the amount of concrete that a human beings use throughout the world. So, what we did is we fundamentally went after the composition of concrete. Now, this is not sexy. It's It's not great to talk about the composition mix of concrete uh when you build when you're building AI infrastructure, but it matters. You pour concrete, it has to cure, it takes time to strengthen, and if you get cracks in it the wrong way, you have to redo the whole process. And you can't to to do that. So, what we did is we built this AI tool that Well, Okay, I'm going to finish my concrete slide. The Uh and we built a AI tool to rethink how sort of concrete works. And the new concrete that we ended up getting has 19% less carbon, cures 43% faster, and is completely cost neutral. So, it's giving back to the uh planet. It's faster for us. And guess what? We open-sourced both the tool and the concrete mix. All of you can go and use it. Use it in your data center. Use it in your backyard. It's good for the planet. Okay. Assumption three was that once it's built, it just works. Now, the thing that I want to emphasize is that AI training clusters is fundamentally a distributed systems problem. I'm a software person uh fundamentally, right? And I think about things in distributed systems. And if you take something that's highly reliable and take a second one that's equally reliable and put together a bunch of it, the end product is not necessarily as reliable as each one of that was. I'll explain. Say you take a component that fails one in five years. That's the rating. I take a second component, right? Now, that fails once in five years, too. Now, if I take a thousand of them, there's something failing every few weeks. Now, if I take a hundred thousand, if I take a million of them, there's something failing every day, every hour. That is That is sort of the semantics you have to uh run with when you're building a mammoth cluster because that's the amount of components that are going into the cluster. The second thing to call out is a sort of new product introduction. Hopefully, this is super familiar to you. That's an Nvidia GB200 mammoth machine, 15,000 lb, three three racks of cooling, one one of This is one of our racks for for what it's worth. Uh and and then see the networking and and the GPUs. The thing that you have to understand here is is that every new system, the more variables you change, the more complex it becomes. So, here we change the data center layout, the cooling, uh the racks, the design, even the concrete. And each one of this adds complexity, and complexity compounds. Now, what you have to realize also is is that when you're creating a cluster, you usually should have a target of what is allowed in terms of interruptions. At this scale, every single failure can either slow down your training run or actually stop it completely. And you won't obviously don't want to do that. So, our North Star was that we wanted to have about two interruptions per 8,000 GPUs every single day. That's sort of our North Star. And to get there, as you can see, you really had to work on it a lot. And I have to give a huge shoutout to all our vendors because they were in it with us a long, long time. And it took us a long time to get this, but I'm very happy to say that we made it here. Uh see, at this stage, what you have to realize is you have to plan for failure. You have to model it, you have to log it, you have to detect it, and you have to work around it. That's the only way you can scale things of sort of this magnitude. Uh this is where I actually want to give a huge shoutout to all of you. All of you here are partners, our vendors. Meta actually thinks about this as a very portfolio-based approach. We don't think a single company can go and do this all by ourselves. When we build data centers, we have design partners, construction partners, partners that run the data center with us. We do similarly, we design networks, the same same for racks, uh same for GPUs, uh same for same for uh the end-to-end system. Uh I my big thing, my big ask to all of you is that make sure that whatever you do and whatever you do in partnership, give back. The more you this I remember being in OCP back in 2011, 2012 when we started it. It was a small gathering of people with the right mindset and it's it's amazing to watch how big this has become and how global it has become. But it was basically born of the of this philosophy that we can advance the whole industry if if we go and act with each other, not sort of against each other. So hopefully if there's one thing you take away from this talk, co-design, co-build. Okay. The last assumption is that software is the easy part. Now you might be thinking what is this I'm a software guy I was saying this, right? What is a software guy doing at a hardware conference essentially? Well, I'm here to be humble and tell you that I thought software was the easy part, but it wasn't. Right? See, the one thing you need to understand is that all the things I've been talking about, there's ten, there's networks, there's reliability, there's sort of a bunch of physical concrete whatever, this is all complexity that exist, but software is the thing that goes, gets all of it together, puts it together into a usable interface and present itself as one single machine. That is the job of software. If I'm a researcher sitting in the lab or if I'm an engineer, he or she is sitting in front of a prompt, typing in commands, and they expect it to magically just work. It's our job to obfuscate away the complexity that exist. Uh one thing I do want to a lot of you work in infrastructure at the end of the day, right? Listen, infrastructure to an extent is like electricity or like water. When you flick on the switch, you want the light to come on. When you open the tap, you You water to flow. People notice only when it does not happen. People rarely notice it when it actually happens the way it is. But that's the job I think all of us signed up for. Complexity we obfuscate away. We just make things work like that. And to the extent that we do that, I think we'll have super successful careers, right? Total aside, not part of my pitch at all. Okay. Uh now, the software that goes into training something at the end of the day is pretty complex as you can imagine. And uh like I was saying, a single thing a single interruption can go and break it up. So, what did we do? Well, we sort of made it like it was a matrix. And what do what do I mean by that? Well, the idea is that you end up building a fault tolerance layer that is both model aware, parallelism aware, and partition aware. To the extent that you do that, then you can detect when the errors happen, route around it, and the model continues to run. And if an error does happen, well, you just have to rerun that part of the job. Nothing more, nothing less. The other thing to call out is that we checkpoint and we checkpoint continuously. So, if you have to roll back, you just have to roll back the last 5 seconds, maybe the last 8 seconds of the job, to the last checkpoint, and then you roll it forward at the same time. Now, remember, we are also training at the same time. We're train training, and we're also adding machines to the cluster again at the same time. So, what we do is the training runs continue. And then what we do is as and when the new machines, the new racks, the new clusters come on, they're added on, and in the next checkpoint, you just incorporate it into the training run, and the cluster just magically becomes bigger. This is sort of the obfuscation of complexity that I was talking about that if we do it right, the developers never even know. The last thing I want to call out is Think back to your time when you were a kid in school, okay? Bear with me. I have a point here. Right? We used to do science experiments, right? And it used to be simple. You You ran an experiment, you jotted down a bunch of results. Then you went and changed a single variable. Jotted ran the experiment again, jotted down the results again. You isolated the cause the effect. That's the process of scientific disco- discovery, and this is how sort of people have done work. We do not have that luxury in real life. I wish. So, in Prometheus, we went and changed a bunch of variables all at the same time. That's the real life that I think all of us live in. It's a huge multivariate experiment that we end up running because we don't have the time, but there are lessons. There are lessons that we take and then we apply it to the next cluster. Hyperion is our next cluster. It's in RPL, Louisiana, Richland Parish, Louisiana, and that's going to be absolutely mammoth. It's a 5-gigawatt cluster under at a single location. To the best of my knowledge, it's the biggest biggest data center on the planet. And it's it's about 4,000 acres last I looked. Uh it's uh it's absolutely hard. It's about the size half the size of Manhattan to give you a sense of scale. That's how big a single data center is. As you can imagine, I've been talking about Prometheus. So, if you can think about all the lessons that we'll probably learn in sort of RPL, that that that'll be a fun talk to give someday. But finally, I want to close about why this matters. Why do we end up going and building sort of this all of this big sort of clusters, right? Is it fun to talk about big numbers, petaflops, etc.? But what matters is sort of what runs on this, right? At the end of the day every call every message like every interaction, every post you do on one of the properties, that that has real meaning to someone. Right? And to the extent that you can go and do that and do that in a sort of good fashion, that is actually the real payoff of all the work that sort of I was describing here. Yeah, this I mean it's awesome to have big vanity numbers, right? A gigawatt a gigawatt, thousands, hundreds of thousands of GPUs, etc. But I think our job at the end of the day, like I was saying, is to give back to the community and hopefully our our businesses and the world a lot more efficiently and a lot more responsibly. That's why we build. Thank you, folks.