Can AI automate real jobs? Epoch tested six frontier models on its own work

Most AI job claims rest on benchmarks. Epoch AI tried something plainer: it gave six frontier agents real tasks from its own work and checked what came back. The answer, published October 8, 2026 by Kelly Hong and Greg Burnham, is that current models cannot yet replace workers, "at least not at Epoch." The details are more useful than the headline.

How the test worked

Epoch paired six models with their own coding harnesses and top reasoning settings: GPT-6 Astra in Codex, Claude Fable 5.1 in Claude Code, Grok 4.6 in Grok Build, Gemini 3.8 Flash in Antigravity, Kimi K3 in Kimi Code, and Qwen 3.8 Max in Qwen Code.

The suite had 11 tasks in five categories: graphic design (3 tasks), data insight generation (3), data explorer generation (2), AI data center research (2) and research design (1). Each model got full permissions in its own workspace and Google account, plus tools such as Figma and reference material, "as much context as we would to a new hire."

Rubrics were built with Epoch staff and mixed objective checks, such as word limits, with subjective ones, such as clear titles. One human grader reviewed each output, and each model ran once per task. The authors say they put more weight on qualitative observations than on the scores.

What the models did well

Fable 5.1 and GPT-6 Astra were broadly tied in the lead. Their strength was reliability on well-defined work such as coding and computational analysis.

Computer use improved noticeably. Earlier models failed to port an article to Substack. The newer ones succeeded. And design is already moving: Epoch removed a restyling task from the suite because Fable 5.1 produced output at Epoch quality, and the team now largely automates that step. That is a concrete, in-house example of a task leaving the human column.

Where they fell short

The failures cluster around judgment, and they are specific.

• Implicit conventions. For a model performance chart, Fable 5.1 produced a far more information-heavy graphic than the simple chart an Epoch designer made, and used "A" and "B" labels as one-off markers instead of the consistent system in the reference example.
• Flawed experiments treated as findings. GPT-6 Astra's pilot gave models a 4096-token output limit, which cut off 61 of 280 responses before they named an input. After raising the budget and seeing improvement, it called the result sensitivity to the acquisition budget. The authors call that misleading, since the truncated cases produced no answers at all.
• Vague designs. Astra proposed a promising research question: "When an AI agent fails to improve with practice, has it failed to collect useful evidence, or failed to understand evidence it already has?" Its experiment never explained what "matched externally selected experience" meant or how its reference bounds would be used.
• Verbosity. Fable 5.1 wrote dense executive summaries and wordy captions, against Epoch's principle of clear, minimal communication.
• Convergence. Fable 5.1, Gemini 3.8 Flash and Grok 4.6 all chose the same polling topic for a data insight, and Kimi K3 and Qwen 3.8 Max wrote similar self-evaluation proposals.

The open-weight gap, and what it says about benchmarks

Open-weight models lagged further behind, even on well-defined tasks. Kimi K3's insight about arXiv rested on a filtering error: it analyzed all arXiv papers rather than only physics papers and never checked its filter. Its diagram of wealth redistribution did not follow the described flow of returns from capital assets.

Yet Kimi K3 scores 158 on the Epoch Capabilities Index, roughly tied with Grok 4.6. The authors argue that this shows existing benchmarks miss real weaknesses. A score that ties two models can hide a large difference in whether you would trust the output unsupervised. For the broader problem, see our explainer on what AI coding benchmarks actually measure.

Three October readings on AI doing AI work

This test lands alongside two other recent pieces of evidence, and they point the same direction. The State of AI Report 2026 cites Anthropic's own internal index, saying Claude led about 26 percent of its measured model R&D work in August, up from under 1 percent in February, with researchers still setting tasks and supervising. And Epoch's InnovationEval asked agents to devise a real post-training improvement and found, in its own words, early evidence that the answer is no. We covered it in Epoch's InnovationEval.

Put side by side: models can do a growing share of measured, well-specified work, and they still struggle to originate good ideas or to catch their own mistakes. Both are consistent with the Epoch conclusion that closed-weight frontier models are reliable on defined tasks and consistently fall short on judgment.

The limits of the test

Epoch lists its own caveats, and they are real. The sample is small, with few tasks and one run per model per task, so scores are noisy. Grading is subjective, and the authors treat the numbers as less informative than the qualitative findings. The tasks reflect Epoch's work and standards, not all work. Some gaps may reflect the agent harness rather than the model. The suite differs from GDPval, AutomationBench, the Remote Labor Index and CRUX by requiring models to gather context with tools in messy, open-ended tasks.

Our take

The finding to carry away is not that AI cannot do jobs. It is that the part of a job AI does best is the part that is easiest to specify, and the part it does worst is knowing when its own output is wrong. We would use these agents on bounded tasks with a human checking conclusions, not on open-ended research. Watch Epoch's reruns: if the judgment gap closes with the next model generation, this article will need correcting.