RobotWorld benchmark: the best frontier model solves 16 of 84 robot tasks

A new benchmark gives a clear answer to a question that gets asked every time a frontier model improves: can it drive a robot yet? In RobotWorld, the best model, OpenAI's GPT-6 Astra, completed 16 of 84 simulated robot tasks. Anthropic's Claude Opus 5.5 completed 13. Three other models managed one or two each, and 63 tasks were solved by no one.

What RobotWorld tests

The paper, "RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments," was posted to arXiv on October 7, 2026 by a team led by Chenxin Li. It describes a simulation testbed for what the authors call "robot use": an agent reads instructions and observations, then acts through a robot's interfaces.

The 84 tasks come from 20 source projects and break down as 38 manipulation, 20 mobile manipulation, 11 locomotion, 11 driving and 4 aerial control tasks. Success is decided by an executable checker, not by how the run looks or what the agent claims. A stacking task, for example, needs the full color order, and a payload-hover task needs every bound to hold for the final two seconds. A run that succeeds after the scored step budget ends counts as a failure; the paper records one Opus 5.5 stacking run that way.

The results

Each of five models got each task once, for 420 scored episodes.

• GPT-6 Astra: 16 of 84, or 19.0%, at about $9,913 and 941 million tokens.
• Claude Opus 5.5: 13 of 84, or 15.5%, at about $2,916 and 1.63 billion tokens.
• Kimi K3: 2 of 84, or 2.4%.
• DeepSeek V4.1 Flash: 1 of 84, or 1.2%.
• Gemini 3.8 Flash: 1 of 84, or 1.2%.

The paper's cost and token figures are estimates from its project accounting, not invoices, and Kimi's are extrapolated from 57 valid records. Across the five, 21 tasks were solved by at least one model. That leaves 63 unsolved by all of them.

The two leaders shared only 8 solved tasks, so they are good at different things. Astra did better on manipulation, 9 of 38 against Opus 5.5's 7 of 38, and the paper says Astra "succeeds more often on spatial and constrained-contact goals." Opus 5.5 did better on aerial control, solving 3 of 4 tasks, against Astra's 1.

Speed and cost differ as much as accuracy. The paper's median elapsed time per run was 30.9 minutes for Astra and 54.0 minutes for Opus 5.5, against 67.6 minutes for Kimi K3, 16.0 for DeepSeek V4.1 Flash and 12.8 for Gemini 3.8 Flash. Fast runs were not good runs: the quickest two models solved one task each. Opus 5.5 used more tokens than Astra, 1.63 billion against 941 million, but cost about a third as much on the paper's accounting.

Some domains were close to empty. Astra and Opus 5.5 each solved 2 of 11 driving tasks, and in mobile manipulation Astra solved 4 of 20, Kimi 1 and Opus 5.5 none. In locomotion, only Opus 5.5 scored, with 1 of 11.

Why they fail

The authors' diagnosis is the useful part. In their words, the models' perception and control capabilities "do not consistently compose into successful behaviour." The paper lists recurring failures: losing track of task-relevant object states, failing to correct ineffective actions, recovering too late, and wrongly judging unfinished tasks as complete.

The traces show how that looks. In one Kimi pouring run, the bottle toppled, repeated grasps failed, and the 400 step horizon ran out. A DeepSeek computer-use run made no robot requests at all and hit the non-action limit during analysis. In a charger-plugging run, a rotation was rejected because the idle arm was in the way, and lowering that arm first let the retry succeed.

That last case is worth noting: the model could fix the problem once it understood it. The weakness is the loop of acting, checking and correcting over hundreds of steps.

How much to read into it

The caveats are the authors' own, and they matter. Each result is a single episode, so the paper's intervals describe variation across tasks, not repeat-trial noise. Comparisons between models are observational, not controlled, and provider differences and collection timing may confound them. The success and failure examples are selected for illustration, and some logs are incomplete. This is a preprint, so it has not been peer reviewed.

It also tests general-purpose models steered through interfaces, not purpose-built robot policies, which is a different claim from "robots are not ready." For how benchmark numbers like these get built and where they mislead, see our guide to what coding benchmarks actually measure; the same cautions about task selection and single runs apply here.

Our take

RobotWorld is a useful reality check on capability talk. A model that tops general-intelligence leaderboards solved under a fifth of these tasks, and most of the benchmark is untouched. That fits the authors' reading that the gap is composition, not any single skill.

We would not read the Astra versus Opus gap as a ranking: 16 against 13 on single runs is within what task selection alone could produce. What to watch is the unsolved 63. If a future model starts solving tasks that all five missed here, especially in mobile manipulation where only Astra and Kimi scored at all, that is real progress. Until then, claims that language models are about to run physical robots should be read against this table.