AI hallucination rate: why one model can score 3% and 23% depending on who measures

An AI hallucination rate is the share of a model's outputs that contain made-up or unsupported content, and it is not one number. Vectara's public leaderboard, updated October 22, 2026, lists the best summarizing model at 1.8 percent, while Moonshot AI reported 23.4 percent for its previous model on its own hard test the same month. Both are honest numbers. They measure different tasks.

If you have seen a headline claim that a new model "hallucinates less," this is how to read it.

What counts as a hallucination

In practice researchers use two definitions. The first, grounded or "closed-book to the document" hallucination, asks whether a model added information that is not in the text you gave it. The second, factual hallucination, asks whether a claim the model made from memory or after searching is false. They are different problems and need different tests.

Neither means the model is lying. Language models generate plausible text; when the right answer is unclear to them they can produce a confident wrong one. That is why the number depends so heavily on the setup.

The summarization yardstick: Vectara

Vectara's leaderboard gives each model source passages and asks for a short summary using only the passage. Vectara's evaluation model, HHEM-2.3, scores whether each summary is consistent with its source. The hallucination rate is 100 minus the factual consistency rate. Short or refused answers are filtered out, and an answer-rate column shows how often a model produced a summary at all. The dataset is private and covers more than 7,700 articles from news, technology, science, medicine, law, sports, business and education, ranging from about 50 to 24,000 words.

The five lowest rates on the board at its October 22, 2026 update were:

• antgroup/finix_s1_32b: 1.8 percent
• openai/gpt-5.4-nano-2026-03-17: 3.1 percent
• google/gemini-2.5-flash-lite: 3.3 percent
• microsoft/Phi-4: 3.7 percent
• meta-llama/Llama-3.3-70B-Instruct-Turbo: 4.1 percent

Notice what is in that list: small and older models. Summarizing a given passage is an easy task, and models trained to stay close to the text do well. A low score here says little about how a model behaves when it must answer from memory.

The hard stress test: Moonshot

Moonshot's Kimi K3.1 Pro announcement shows the opposite end. It reports a claim-level hallucination rate on its own "Hallu Bench Hard," with tools, of 23.4 percent for Kimi K3 and 8.3 percent for K3.1 Pro. In the same table GPT-6 Astra is 5.5 percent, Claude Opus 5.5 is 15.0 percent and Claude Fable 5.1 is 23.3 percent. Without tools, GPT-6 Astra is 4.2 percent, K3.1 Pro 5.4 percent and Opus 5.5 6.8 percent.

Moonshot is explicit that this is a stress test built to expose errors and that the rates "do not represent overall error rates in daily use." So a model can look like a 3 percent hallucinator on one yardstick and a 23 percent one on another, with no inconsistency. Moonshot also grades its own test and chooses the competitor versions it compares against. The same launch is covered in our report on Kimi K3.1 Pro.

Why models guess: the Kalai paper

A September 2025 paper, "Why Language Models Hallucinate," by Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, offers a structural reason. It argues hallucinations occur because training and evaluation procedures reward guessing over expressing uncertainty. If a benchmark gives credit for a lucky guess and none for "I don't know," a model tuned to the benchmark learns to guess. The authors describe this as an "epidemic" of penalizing uncertain answers and propose changing how existing, widely used leaderboards are scored, rather than adding more hallucination-specific tests.

That is a claim about incentives, and it is from a preprint, not a settled result. But it explains an observation anyone can make: models rarely say they do not know unless their scoring makes that the better move.

A buyer's checklist

When you see a hallucination figure, ask five questions:

• What task was it? Summary of a given document, answers from memory, or answers with search.
• Were tools allowed, and which?
• Who wrote the test, and who graded it? A model grading models is itself imperfect.
• Was abstaining allowed or rewarded, and how many questions did the model decline?
• What date and which model version? Rates change with every release, and labs revise their tests.

Our guide to what AI leaderboards rank and what a top score hides applies here too, as does the way benchmarks saturate: a number that once separated models stops doing so.

One model, three numbers

Moonshot's own table shows how much the setup matters. Claude Opus 5.5 scores 6.8 percent without tools and 15.0 percent on the hard test with tools. GPT-6 Astra scores 4.2 percent and 5.5 percent. Claude Fable 5.1 scores 11.4 percent and 23.3 percent. The ordering is not the same in both columns, and the gap between the columns is larger for some models than others. That is why a single hallucination percentage in a launch headline tells you almost nothing without the conditions behind it.

Our take

Hallucination is worth measuring and worth treating skeptically. The practical guidance is dull but reliable: for tasks with a source document, hallucination is now low for most current models. For open questions from memory, assume a real error rate, check anything that matters, and prefer systems that show sources or decline to answer. No leaderboard number replaces testing the model on your own questions.