Rack-scale AI explained: why Nvidia and AMD now sell 72-GPU racks instead of chips

Rack-scale AI is the idea that the thing a lab buys is no longer a graphics chip but a whole rack, designed as one machine. Nvidia sells it as the GB200 NVL72. AMD's answer is called Helios. Both put 72 accelerators in a single liquid-cooled rack, and both claim the rack behaves like one very large GPU. The details explain why training clusters now cost what they do, and why the vendors' headline numbers need careful reading.

What a rack-scale system is

Older AI servers were boxes of eight GPUs, linked to other boxes over slower networks. The cost of that arrangement is communication: when a model is too big for eight GPUs, the chips spend time waiting on each other. A rack-scale system moves the boundary. Nvidia's product page describes a 72-GPU NVLink domain that works as a single large GPU. The rack holds 36 Grace CPUs and 72 Blackwell GPUs, built from superchips with one Grace CPU and two Blackwell GPUs each.

Nvidia lists 130 TB/s of NVLink bandwidth for the rack, or 1.8 TB/s per GPU from fifth-generation NVLink. GPU memory is 13.4 TB of HBM3E, and CPU memory is 17 TB of LPDDR5X. The rack is liquid-cooled, which the page says increases compute density and reduces floor space. For scale-out between racks it uses Quantum-X800 InfiniBand, Spectrum-X800 Ethernet and BlueField-3 DPUs.

What AMD's Helios contains

AMD's version is a reference design, not yet the deployed rack. StorageReview, in an article published July 23, 2026, reports Helios holds 72 MI455X GPUs, 18 Venice CPUs and 12 Broadcom Tomahawk 6 switches in the Open Rack Wide format. Rack totals are 2.9 exaFLOPS of FP4 compute, 31 TB of HBM4, 260 TB/s of scale-up bandwidth and 43 TB/s of scale-out. Power is roughly 225 to 245 kilowatts per rack, with the GPUs connected over UALink carried on Ethernet.

Each MI455X has 432 GB of HBM4, per the same article. StorageReview also puts Nvidia's Vera Rubin NVL72 at 20.7 TB of HBM and the same 260 TB/s of rack-level scale-up. Those Rubin figures are from the article's comparison and describe Nvidia's next generation, not the GB200. [[INLINE]]

The numbers that are marketing

Several headline figures come with a trap attached.

• "30x faster" inference. Nvidia's page says GB200 NVL72 delivers 30 times faster real-time trillion-parameter LLM inference than H100, and marks it as projected and subject to change. It also claims 4x training speed against H100, based on a test comparing 456 NVL72 GPUs with 4,096 H100 GPUs, and 25x more performance at the same power. These are comparisons against a two-generation-old part.
• Sparse versus dense. Nvidia's footnotes say tensor-core figures are sparse and dense figures are half the sparse number. A reader who quotes a 1,440 petaflops number without that footnote has doubled the real figure for ordinary workloads.
• Different FP4 formats. StorageReview notes AMD's MXFP4 and Nvidia's NVFP4 use different scaling, so cross-vendor FP4 comparisons only mean something at the application level, such as tokens per second at matched accuracy.
• Reference designs. The same article says Helios figures are a reference design and deployed systems can differ significantly. Its author wrote: "Those are AMD's numbers against NVIDIA's published numbers, not independent measurements."

The honest summary is that the architecture is documented and the speed-ups are claims.

Why the rack became the unit

Three pressures pushed the industry here. First, models grew past what one server can hold, so the link between chips matters as much as the chips. Second, power and heat: at 225 to 245 kilowatts a rack, air cooling is not an option, which is why liquid cooling appears on both vendors' designs. Third, buyers want one supplier to answer for the whole system, including networking, rather than integrating parts themselves.

That has a financial consequence. When the product is a rack, a customer's purchase is bigger, more concentrated and harder to unwind. Our Big Tech capex tracker shows how much cash Alphabet, Amazon, Meta and Microsoft are putting into this kind of hardware, and our explainer on how Google's TPUs differ from Nvidia GPUs covers the third architecture in the race.

Who is adopting AMD's rack

CNBC reported on November 3, 2026 that OpenAI, Meta, Microsoft, Anthropic and Oracle are beginning to use AMD's Helios rack-scale systems in data centers, with no OpenAI-specific details given. That matters because a second rack-scale supplier gives buyers leverage. It does not yet prove performance parity. StorageReview says Nvidia keeps advantages in the host link, the data processing unit and the software ecosystem, and that real-world performance will be known only when racks reach hyperscaler deployments.

What to check before believing a rack-scale claim

• Is the baseline a current part or an H100?
• Is the figure sparse or dense, and which number format?
• Is it a measured result on a named model, or a projection?
• Is it a reference design or a rack running in a customer's building?
• Who ran the test: the vendor or a third party?

Our take

Rack-scale is a real shift, not a slogan. A 72-GPU domain changes what models can be trained and served efficiently, and both vendors document it. But the performance multiples you will see in headlines are vendor projections against old baselines, and the cross-vendor comparisons are not apples to apples. Use the architecture facts, which are solid, and wait for independent tokens-per-second results before accepting the rest.