Detailed view of a green electronic circuit board with microchips and components
ExplainersIndependently verified

Agentic inference capacity explained: why a GB300 is worth 25 H100s and not 2.5

By Himanshu Sakre·

Photo: Ivan Chumak / Pexels

Epoch AI's new Inference Performance Index says memory and chip links, not raw arithmetic, decide how many AI agents a chip can serve. Here is how the measure works and where it can mislead.

Agentic inference capacity is an estimate of how many AI agents, or users, a set of chips can serve at the same time. Epoch AI introduced a way to measure it on November 9, 2026, called the Inference Performance Index, and the headline result is striking: it estimates the Nvidia GB300 provides about 25 times an H100's capacity per chip on agentic workloads, with only 2.5 times the 8-bit floating-point operations per second.

That gap is the whole point. For years, chip comparisons leaned on FLOP/s, the raw arithmetic a chip can perform. Serving a long-running coding agent leans on other things.

Why arithmetic is not the bottleneck

Language model inference has two phases. Prefill processes the input and stores intermediate results in a key-value cache. Decode produces the response one token at a time, each pass reading the cached context and the model's weights from high-bandwidth memory. Epoch's example is a coding agent that sends 200,000 tokens of cached history plus 1,000 tokens of new code-search results, then generates 1,000 tokens. Only the new input needs prefill, but the whole cache stays in memory and gets read again.

So the limits are how much memory a chip has, how fast it moves data, and how quickly chips talk to each other. Epoch's report says that is why raw FLOP/s understates the GB300's advantage by a factor of ten.

The tradeoff the index is built around

A chip can serve many requests slowly or few quickly. Epoch gives rounded values: raising concurrency from two to eight requests per chip lowers output speed from about 210 to about 80 tokens per second per request. Total throughput still rises, from roughly 420 to 640 tokens per second per chip, about 50% more, even though each user waits longer.

The index handles this by counting how many simultaneous requests a chip can support across a range of required output speeds. Version 1 focuses on agentic workloads and combines results across four representative models. Scores are expressed in GB300 equivalents: the GB300 scores one by definition, and a score of two means a chip serves twice as many users or agents at once on average.

“memory and inter-chip communication can matter more than arithmetic throughput for inference performance.”

Jason Li, Epoch AI, Inference Performance Index report, November 9, 2026
Detailed image of microchip components viewed through a microscope lens
A circuit board with a processor: memory and links, not only arithmetic, set how many agents it can serve. Photo: Tima Miroshnichenko / Pexels

Why system design matters

Chips do not work alone. Epoch estimates a Nvidia GB300 NVL72, which links 72 GPUs through high-bandwidth connections, has about 1.9 times the modeled inference capacity per GPU of a B300 HGX, which links groups of eight, even though both use the same Blackwell Ultra GPUs. The takeaway is that the rack, not just the chip, is the unit that counts.

To estimate performance, Epoch uses Nvidia's AISimulate tool with inputs for arithmetic throughput at three precisions, memory capacity, memory bandwidth, interconnect bandwidth and the grouping of chips. Its results are analytical projections of optimal performance under the modeled hardware limits, which the report calls speed of light estimates.

What the index found about the market

On November 18, Epoch applied the index to chip sales data. It estimates the agentic inference capacity of AI chips sold grew 4.6 times between the second quarter of 2025 and the second quarter of 2026, while peak 8-bit computing power grew 2.6 times. Epoch's chips overview adds context: the five largest chip designers had shipped roughly 20 million AI chips by late 2025, Nvidia accounted for about half of units, and a single chip draws around 1,000 watts at full load.

Those figures help explain why spending and capacity can diverge. Our report on Anthropic's compute bill shows spending up about 4x a year. Our earlier analysis of how many agents HBM supply could support shows the same memory limits from the supply side, and our compute explainer covers training costs.

Where the index can mislead

Three cautions apply. First, these are simulated, best-case numbers, so a real cluster with scheduling overhead, failures and uneven traffic will deliver less. Second, the index models agentic workloads with long cached contexts. A chatbot with short prompts, or a batch job, could rank chips differently. Third, a score is relative to GB300, so a new reference chip would shift every figure.

What the index implies for the supply chain

A common unit lets analysts add up capacity across chip families. Epoch says the index can estimate a data center's total inference capacity, or even the world's, and combined with performance estimates for a given model and output speed, gives a rough count of how many users or agents hardware could serve at once.

Epoch's chips overview supplies the market backdrop. Nvidia's flagship chip price rose from $5,700 for the P100 in 2016 to $34,000 for the H100 in 2022, yet the H100 performs seventeen times as much computation per dollar. Nearly all leading-edge AI chips are made by TSMC, mainly in Taiwan, and high-bandwidth memory from SK Hynix, Samsung and Micron, along with ASML's EUV lithography machines, are the other main chokepoints. If memory and links drive agent capacity, then the memory makers and the rack-scale designers matter more to the agent economy than FLOP/s tables suggest.

Our take

The index is useful because it corrects a habit. When someone says a new chip is 2.5 times faster, ask faster at what. For agents, the answer is increasingly memory and links, which is also why high-bandwidth memory supply keeps turning up as a chokepoint. Use the index to compare generations and designs, and ask any vendor claiming a capacity figure which workload, which output speed and which precision they assumed.

Frequently asked questions

What is agentic inference capacity?

It is an estimate of how many AI agent or user requests a chip, rack or data center can serve at once. Epoch AI measures it with its Inference Performance Index, in GB300 equivalents, across long, cached-context workloads.

Why is a GB300 worth 25 H100s for inference?

Epoch estimates about 25x the H100's capacity per chip on agentic workloads, though only 2.5x the 8-bit FLOP/s. Memory capacity, memory bandwidth and chip-to-chip links matter more than arithmetic for serving long contexts.

What is the Inference Performance Index?

It is Epoch AI's index, introduced November 9, 2026, that compares chips by how many simultaneous requests each can serve across a range of output speeds. Version 1 covers agentic workloads and combines four representative models.

How fast is agentic inference capacity growing?

Epoch estimates the agentic inference capacity of AI chips sold grew 4.6x between Q2 2025 and Q2 2026, versus 2.6x for peak 8-bit computing power.

Are the index numbers real-world measurements?

No. Epoch uses Nvidia's AISimulate to project optimal, speed-of-light performance under modeled hardware limits. Real deployments typically deliver less because of overheads, failures and uneven traffic.

What is the difference between prefill and decode?

Prefill processes the input and stores it in a key-value cache. Decode generates the response token by token, using the cache and the model weights. Epoch models both and provisions enough prefill chips not to bottleneck decode.

Sources

What each one is, and whose it is.

  1. OtherIndependent of the vendor
  2. DatasetIndependent of the vendor
  3. 3

    What you need to know about AI chips, Epoch AI (November 18, 2026)

    OtherIndependent of the vendor