When every model scores 100 percent, the score is useless. A new paper says the working out is not.

A benchmark dies by being passed. Once the leading models all score 98, 99 and 100 on the same test, the test stops telling you which one is better. The number has run out of room.

That is not a scoreboard problem. It is a safety and policy problem, because benchmark scores are load bearing. They appear in system cards, in safety cases, in procurement decisions and in draft regulation. A measure that cannot separate a very capable model from an extremely capable one cannot support any of that.

A preprint posted to arXiv on October 14, 2026 makes a simple proposal for getting more life out of saturated tests. The paper is "Evaluating Model Capabilities After Benchmark Saturation," by Morio Hamada, Jehyeok Yeon, Matteo Migliarini, Justin Olive and Magda Dubois. It has not been peer reviewed.

The idea

When a model solves a benchmark question correctly, the grader records a one and throws away everything else. But the model did not just produce an answer. It produced a transcript: reasoning, intermediate steps, tool calls, the whole path to the result.

Two models can both get a question right and get there very differently. One reasons cleanly in three steps. The other flails through twelve, calls a tool it did not need, and arrives in the right place anyway. The score says they are identical.

The authors' claim is that those discarded transcripts still carry information about how capable a model is, even when every transcript in the set ends in a correct answer.

How they tested it

Rather than wait for a benchmark to saturate, they went backward. They used historical benchmark and model releases to simulate saturation retrospectively, which lets them check the method against capability levels that are already known.

The test is whether protocols that look only at correctly solved transcripts from a saturated benchmark can predict how a model performs on held-out items that are not saturated. If they can, the saturated benchmark still has signal in it.

They compared several approaches: judging transcripts against each other in pairs, scoring a single transcript on its own, and predicting from deterministic features of the transcript or from embeddings.

What they found

Saturated transcripts do retain capability information beyond the benchmark score.

Pairwise judging worked best. Measured against a baseline that uses only the saturated benchmark scores, it reduced prediction error by 5.3 times.

The useful signal is spread across the transcript rather than sitting in one place, but it concentrates in particular channels, and the two the authors name are reasoning and tool calls. That is intuitive. Those are the parts where a model's approach shows, as opposed to the final answer, where every correct response looks the same.

The limitation the authors print themselves

This is the part worth taking seriously, and it is to the paper's credit that it sits in the abstract rather than buried in a appendix.

The predictive power does not always survive a distribution shift. The example they give is specific: these protocols underestimated the capabilities of the first reasoning models introduced in early 2025.

That is exactly the failure mode that matters. A method for extending benchmark life is most needed at the moment something genuinely new arrives, and this one was least reliable at precisely such a moment. A technique that works smoothly until the architecture changes, and then misleads you, is not obviously safer than admitting the benchmark is dead.

Why this matters beyond the leaderboard

We have written before about what a top leaderboard score hides. This is the next stage of the same problem: what happens when the score hides everything, because every score is the same.

The practical reading is that the evaluation community is running out of headroom faster than it can build new tests, and is now asking whether existing tests can be squeezed harder. Mining transcripts is a reasonable thing to try. It is also, unavoidably, harder to audit than a number. A benchmark score can be checked by anyone with the test and the model. A pairwise judging protocol over transcripts introduces a judge, and a judge has its own biases and its own failure modes.

If evaluation moves this way, the question a regulator or a safety reviewer has to ask gets harder, not easier. Not what did it score, but who judged the transcripts, and how do we know the judge was right.