Meta details the Prometheus supercluster: phase 1 complete, with reliability data
Meta says Prometheus, its supercluster in New Albany, Ohio, finished phase 1 and trained Muse Wonder; the target is about 1 million GPUs.
Prometheus is Meta’s AI training supercluster in New Albany, Ohio. In an engineering post dated October 19, Meta says the first phase is complete and was used to train its Muse foundation models, including Muse Wonder. The full build is meant to reach roughly one million GPUs, a goal Meta says it is still working toward. The same post reports reliability figures from Muse Wonder’s training runs: pre-training goodput above 85%, and above 90% for reinforcement learning.
Key points
- Meta says phase 1 is complete and was used to train the Muse models, but gives no completion date. The cluster now holds “hundreds of thousands” of GPUs across several buildings.
- Muse Wonder’s runs held pre-training goodput above 85% and RL goodput above 90%. Mean time between interruptions was about 47,000 GPU-hours, and firmware and other fleet changes lifted it above 54,000.
- Interruptions peaked near 1,000 per 8,000 GPUs per day in mid-2025. Meta says it cut that peak by 99% and targets two per 8,000 GPUs per day.
Background
Prometheus has come up in earlier reporting and Meta posts. In January, CNBC reported that Meta had signed nuclear power agreements with Vistra, TerraPower and Oklo, which plan to help power Prometheus eventually, and that Meta expected the cluster to come online sometime in 2026. In February, a Meta engineering post said that once complete, Prometheus “will deliver 1-gigawatt of capacity.” That describes the finished site as a target. Meta’s October post gives no total power figure for it.
When Muse Wonder launched on October 12, Meta said it was “Trained at our gigawatt-scale Prometheus data center.” Our October 13 story on Muse Wonder already reported that link, so it is not repeated here. Meta is not alone in buying nuclear power for AI, and Google’s nuclear deal with Constellation is another example of the same trend.
The facts
Figures come from Meta’s October 19 post unless noted. They are Meta’s own numbers, and the table keeps the post’s qualifiers.
| Item | Figure | Qualifier |
|---|---|---|
| Phase 1 | Complete; used to train Muse models | No completion date |
| Current size | “Hundreds of thousands” of GPUs | Across several buildings |
| Target size | About 1 million GPUs | No timeline |
| Pre-training goodput | Above 85% | Muse Wonder runs |
| RL goodput | Above 90% | Muse Wonder runs |
| MTBI | About 47,000 GPU-hours; above 54,000 after firmware fixes | Wall-clock, in GPU-hours |
| Interruption peak | Nearly 1,000 per 8K GPUs per day | Early phase, mid-2025 |
| Reduction | 99% (Meta, October 19); factor of 50 (September talk, per GamesBeat) | 99% is against the mid-2025 peak, over about a year; the 50x report gives no baseline; not merged |
| North-star target | About two per 8K GPUs per day (about 100,000 GPU-hours MTBI) | Target |
The post defines an interruption broadly: any fault that forces a job to pause and recover, including GPU silicon defects, network fabric anomalies, storage bottlenecks and thermal limits. In the post’s framing, a single-host fault lowers a job’s goodput by slowing or stopping it.
Hardware and power, as Meta states them:
| Platform | Rack layout | Power (Meta’s figure) | Status |
|---|---|---|---|
| GB200 NVL72 (Catalina, Ariel variant) | Two racks, 36 GPUs each | About 120 kW for the pair | Deployed |
| GB300 NVL72 (Clemente) | One rack, 72 GPUs | About 135 kW per rack | Deployed |
| Vera Rubin NVL72 (San Miguel) | One rack, 72 GPUs | Over 170 kW per rack | “Arriving soon” |
- Each Ethernet back-end fabric connects up to 20,736 GPUs.
- Moving from NVL8 to NVL72 widened the scale-up domain from 8 GPUs to 72, and Meta says that raises the fault blast radius ninefold. Each NVL72 domain trains on a 64-GPU slice and keeps eight GPUs as in-rack spares.
- Synchronized training can swing cluster load by 30% to 40%. Dynamo Dimmer lowers GPU power limits gradually as the cluster nears its electrical ceiling, instead of hard-stopping jobs.
What others say
Prometheus’s reliability numbers come mostly from Meta alone, so outside reporting mainly helps with timeline and definitions.
- GamesBeat’s September 23 report covered a talk by Meta’s head of infrastructure, Santosh Janardhan, at the AI Infra Summit. Janardhan said Meta had brought interruptions down “by a factor of 50,” with a target of about two per 8,000 GPUs per day. The report also says Prometheus “remains under construction while serving training jobs and adding GPUs.”
- A factor of 50 is a 98% cut, close to the post’s 99%. The post measures its 99% against the mid-2025 peak, over about a year. GamesBeat’s report gives no baseline or window for the 50x figure, so we list both side by side without merging them.
- The September report still described an active build. The October post calls phase 1 complete but gives no date, so we cannot order the two.
- TechCrunch (October 12) said Muse Wonder was trained at a central Ohio data center that “came online earlier this year.” That is the reporter’s wording, and the post gives no date.
- CNBC (January 9) said Meta expected Prometheus to come online in 2026. Its 6.6-gigawatt figure is the capacity Meta said its three nuclear partnerships should add by 2035. It is not the power draw of the Prometheus site.
- As of writing, we found no independent review of the reliability figures and no published critique. For now, the numbers rest on Meta’s account.
Our take
For teams planning large training runs, the post’s value is that its numbers are specific enough to check against. Three design points matter more than any single figure:
- Redundancy sits inside the rack. Each NVL72 rack trains on 64 GPUs and keeps 8 spares. When a GPU fails, a spare takes its place at once, the job restarts from its last checkpoint, and the rack stays in the job instead of waiting for hardware repair.
- Repairs ship in batches. Meta says it can push firmware and driver fixes to more than 100,000 GPUs within a few hours.
- Power swings are handled in layers: Dimmer adjusts power caps, rack batteries absorb immediate spikes, capacitor trays smooth peaks, and facility storage works at site scale.
For most app developers, the post changes little. Its main use is for people who run or plan training clusters. Three catches matter:
- The definition of interruption is broad. Before comparing with another cluster, check whether that operator counts the same events.
- The 99% is relative to the mid-2025 peak, and the post gives no current absolute interruption count. The 50x figure has no stated baseline, so neither shows the absolute level of reliability.
- One million GPUs is a target without a date, and phase 1’s completion has no date either. The post gives no current total power for the site. “Gigawatt-scale” is the project’s name; the 1-gigawatt figure is February’s target for the finished site.
What to watch
- Prometheus’s timeline for roughly 1 million GPUs. Meta gives none.
- Vera Rubin (San Miguel). The post says it is “arriving soon” but gives no date.
- Hyperion, Meta’s next supercluster in Richland Parish, Louisiana, which the post says is being designed with Prometheus in mind. It also has no timeline.
- Whether Meta publishes its current absolute interruption count, since the post gives only the reduction from peak, and a baseline for the 50x figure.