DeepSeek V4 Pro cyber benchmark: 28 of 32 bugs found, and what pooling hides

DeepSeek V4 Pro 0813 found the most known software vulnerabilities in a cybersecurity test run by Aikido Security, 28 of 32 when three runs are pooled, and it did so for about $295. That is the claim being repeated in an October 25, 2026 write-up. The test is real and its design is public. The headline hides how much of the result depends on pooling.

What Aikido tested

Aikido, a security company, published the study under the title "We burned 11.7bn tokens to find the best cyber AI model." The post is dated August 21, 2026 and was last updated October 5, so the underlying test is two months old, even though coverage is recent. It ran 10 models against 32 recently disclosed vulnerabilities in real-world repositories, three runs per model, 96 runs in all, using about 11.7 billion tokens.

Each model worked from source code in a bounded version of Aikido's code analysis harness, with a 30-turn maximum and no internet access. Prompts, tools, cases and the evaluation policy were frozen across runs. Recall is the number of vulnerabilities found. Precision is the share of reported findings that were accepted as real. A pass is one full run over the 32 cases.

Models tested were GLM-5.3 in preview and final form, DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731, Qwen3.8-Max, Kimi K3, Grok 4.6, Opus 5, Sol and GPT-5.6 Terra.

The numbers, side by side

• DeepSeek V4 Pro 0813: 28 of 32 pooled across three runs, the highest. It found 17 on its first pass, and only 10 were found in all three runs. Three runs cost about $295.
• Qwen3.8-Max: 19 on its first pass, 26 pooled.
• Grok 4.6: 24 on its first pass, 26 pooled, and 21 found in all three runs, the most consistent result.
• Opus 5: 26 pooled, a best single pass of 25, and 19 found consistently.
• Sol: 25 of 32, with 19 found consistently.
• Kimi K3: 17 on its first pass, 25 pooled, with 92.3% pooled precision, the highest among the open-weight models tested.
• GLM-5.3 final: 25 of 32, up from 24 in the preview. Consistency rose from 14 to 18. Cost per vulnerability rose from $14.77 to $19.65.
• DeepSeek V4 Flash: $108 for three runs, 24 pooled.

A single pass by a frontier closed model cost roughly $450 to $590. Seventeen of the vulnerabilities were found by every model at least once, and one was missed by all ten.

Why pooling matters

Pooling means counting a vulnerability as found if any of the three runs found it. That is a legitimate way to measure coverage if you can afford to run a model several times and merge the results, and it is how Aikido makes its point that several cheap runs can beat one expensive pass. But it is not the same as reliability. DeepSeek's own first pass found 17 and only 10 appeared every time. Grok found fewer in total than DeepSeek but repeated 21 of its findings in every run.

So the honest reading has two parts. If you need maximum coverage and can triage many reports, DeepSeek's pooled score is attractive and cheap. If you want a result you can trust from a single run, the consistency numbers say the ranking changes. The Startup Fortune write-up of October 25, 2026 reports DeepSeek at 87.5% and Opus 5 and Qwen at 81.3% each, which is the same data expressed as percentages of 32.

What the test does not show

Thirty-two cases is a small sample, and every model is judged only on vulnerabilities that were already disclosed. Aikido designed the cases to be fresh so that models would reason rather than recall, but we cannot verify that from outside. The test excludes internet access, the harness caps turns at 30, and the findings concern known classes of flaws in source code, not a model's ability to build working exploits. Aikido sells security products, so this is a vendor study. We did not find an independent replication.

Precision also matters for practical use. A tool that reports many findings with a lot of false positives costs analyst time. Aikido reports precision figures in the post, and the Kimi K3 figure of 92.3% is the one we could confirm in what we opened. We could not confirm precision for DeepSeek from the primary page and do not state it.

How it fits the wider picture

Epoch AI's October 16, 2026 data insight places the strongest open-weight model, Z.ai's GLM-5.3, at 160.10 on its Cyber ECI, versus about 171 for the closed frontier models Claude Opus 5.5 and GPT-6 Astra, with Epoch concluding that the open cyber frontier sits roughly where the closed frontier stood six to seven months earlier. Epoch's data is as of October 8, 2026 and rests on a different method, a joint fit across 35 cyber benchmarks. The two studies point the same way: open-weight models are close behind on defensive bug-finding. Our tracker on the open-weight cyber capability gap follows that gap.

The Startup Fortune piece also links the result to Google's Gemini 4 Argon, which the company limited to vetted defenders at launch. It quotes Koray Kavukcuoglu of Google DeepMind: "safely releasing frontier capabilities at this level requires a phased approach." If open-weight models can find bugs at similar rates, gating becomes harder to justify on capability alone. We covered the launch in our report on Argon's cyber-defenders-first rollout.

The hype check

The claim "an open model beat the closed frontier at finding bugs" is true on one 32-case, vendor-run test, on pooled recall, with a precision gap we could not check. It is not evidence that DeepSeek is the better security model in general, and it does not show anything about offense.

What we would do

If you run a security team, use the cost-per-coverage finding: run cheaper models several times and triage the union. Do not drop closed models because of one pooled number. Wait for Aikido's Arena page, which the post points to for newer benchmarks, and for an independent replication before changing a purchase decision.