Meta Details Prometheus, the Gigawatt Supercluster Training Its Muse Models
Meta published a detailed engineering account of Prometheus, its gigawatt-scale supercluster in New Albany, Ohio, that trained the Muse models — with plans to…
By Icarus5 min read
Meta’s engineering team published the most detailed account yet of the Ohio supercluster behind its Muse models — and revealed plans to scale it toward a million GPUs.
Meta’s engineering team has published a detailed technical account of Prometheus, the gigawatt-scale AI supercluster in New Albany, Ohio, that trained the Muse family of models, and revealed plans to scale it toward roughly one million GPUs.
The disclosure, posted to Meta’s engineering blog on October 19, confirms that Prometheus’s completed first phase trained the Muse foundational models, including the Muse Wonder frontier models that Meta says deliver state-of-the-art performance on personal and professional agentic tasks. Prometheus currently runs on hundreds of thousands of GPUs spread across traditional and AI-optimized data center buildings and rapid-deployment structures, all operating as a single machine. Two years before the disclosure, the New Albany site was a conventional data center running web traffic; Meta reworked it for frontier-model training and is building the supercluster in phases.
Two NVIDIA Platforms in Production
Prometheus deploys two NVIDIA GPU platforms, and Meta says a third based on NVIDIA’s Vera Rubin NVL72 is arriving soon. Each uses an NVL72 rack architecture in which 72 GPUs act as a single accelerator within one network fabric.
The first platform, Catalina, uses the NVIDIA GB200 NVL72 in the Ariel two-rack variant: 36 GPUs per rack are joined over NVLink into a unified 72-GPU domain, each Blackwell GPU paired with a Grace CPU and two ConnectX-7 400G NICs at a 1:1 CPU-to-GPU ratio, delivering 800G of I/O per GPU. Meta lists Catalina at 72 Blackwell GPUs with 13.4 TB of HBM3e, 720 peak PFLOPS of FP8 training compute (1,440 PFLOPS at FP4), 130 TB/s of peak NVLink bandwidth, 576 TB/s of peak aggregate memory bandwidth, and roughly 120 kW of total system power across the two-rack pair.
The second platform, Clemente, moves to the NVIDIA GB300 NVL72. Meta says GB300 raises HBM capacity per GPU by 1.5x, from 192 GB to 288 GB, and dense FP4 throughput by a similar factor, while roughly doubling attention acceleration. Clemente packs the full 72-GPU domain into a single rack with a 2:1 GPU-to-CPU ratio and delivers over 1 exaFLOP of peak FP4 compute at about 135 kW of rack power, with a stated 5–10% performance improvement over the GB200 NVL72. Meta’s upcoming San Miguel platform, listed with 72 Vera Rubin GPUs carrying 288 GB of HBM4 and 2.52 peak EFLOPS of FP4 training compute per rack, targets a 10–20% performance-per-watt improvement over the GB300 NVL72.
Built for Fewer Interruptions
Much of the post focuses on reliability, the problem that dominates training at this scale. The move from the H100 generation’s NVL8 architecture to NVL72 expanded the scale-up fault domain from eight GPUs to 72, a nine-fold larger blast radius. Each NVL72 domain runs a 64-GPU slice and keeps eight GPUs as in-rack spares, with roughly 2% additional racks in reserve. For a 32,768 active-GPU training run, that means provisioning 523 NVL72 racks, or 37,656 GPUs, about 13% above the active footprint.
Fault recovery uses two mechanisms: Parallelism-Aware Fault Tolerance takes a faulty data-parallel replica offline while the rest of the cluster repeats the step and continues training, and distributed asynchronous checkpointing with delta checkpointing and in-memory staging restores the job from its last saved state when a fault’s blast radius exceeds one replica. During Prometheus’s initial phase in mid-2025, interruptions peaked at nearly 1,000 events per 8K GPUs daily; Meta reports a 99% reduction after a year of cross-functional engineering, and states its reliability north star is two interruptions per 8K GPUs per day.
Meta reports that it sustained more than 85% goodput on Muse Wonder pre-training runs and more than 90% on reinforcement learning runs, with a wall-clock mean time between interruptions of about 47,000 GPU hours during training, raised to more than 54,000 GPU hours after subsequent fleet-wide changes, including firmware fixes.
Networking and Power
Alongside the standard data center network, Meta operates a dedicated Ethernet back-end network for direct GPU-to-GPU communication within and across buildings. Each building-level Ethernet fabric interconnects up to 20,736 GPUs. The Backend Aggregation network, a multi-layer hub-and-spoke design linking building fabrics across distances of 2 km to more than 100 km, uses Arista 7800-series chassis and delivers multiple petabits per second of bandwidth per region pair. Meta says running pre-training over those distances required a dedicated optimization effort with Meta Superintelligence Labs because of speed-of-light constraints.
Synchronized training triggers power transients of 30–40% within the cluster, and starting or pausing a major job can shift loads of tens of megawatts in seconds. Meta’s Dynamo Dimmer framework gradually adjusts GPU power limits as the cluster nears its electrical ceiling instead of hard-stopping workloads, backed by rack-level battery backup modules, capacitor trays, facility battery energy storage, and coordination with utility providers. The company states it pays the full cost of energy for the New Albany site and matches its energy use with new clean and renewable energy.
Meta is designing Hyperion, its next supercluster in Richland Parish, Louisiana, from the ground up around Prometheus’s lessons, with closed-loop liquid cooling and full grid connectivity with utility partner Entergy.
Why It Matters
The Prometheus disclosure is Meta’s most detailed account yet of the infrastructure behind its recent model releases. Muse Wonder, Meta’s first frontier model family, and Muse Spark Open, its 816-billion-parameter open-weight model, were both trained in this cluster, and the planned scale-up to roughly a million GPUs shows how much training capacity Meta expects the agent era to demand. Muse users’ practical takeaway: the personal agent on their phone sits on top of one of the largest single training machines on earth.
