Recall at a 1% false positive rate, explained: how to read an AI safety monitor claim

Recall at a 1% false positive rate sounds like jargon, but it is the number that decides whether an AI safety monitor is useful or noise. Anthropic's October 16, 2026 research on Claude self-assessment reports 78% recall overall and 94% for high-priority conversations at a 1% false positive rate. Here is what those figures measure, what they hide, and how to read the next lab monitoring claim.

The two numbers a monitor lives on

A monitor reads traffic and flags what looks harmful. Two rates describe how well it does. Google's machine learning course defines recall, also called the true positive rate, as "the proportion of all actual positives that were classified correctly as positives", calculated as true positives divided by true positives plus false negatives. It defines the false positive rate as "the proportion of all actual negatives that were classified incorrectly as positives".

In plain terms, recall asks how much of the bad stuff the monitor catches. The false positive rate asks how much of the harmless stuff it wrongly flags. Tightening one usually loosens the other, so a result is only meaningful when both are stated together. "78% recall" alone says nothing; "78% recall at a 1% false positive rate" is a complete claim.

Precision is a third measure: of everything flagged, how much was truly harmful. Precision depends on how common harm is in the traffic, which is why a monitor with a low false positive rate can still bury reviewers.

What Anthropic reported

Anthropic's paper describes a technique where Claude writes a short safety assessment of its own conversation right after responding. Authors include Yiming Zhang, Nicholas Carlini and Erik Jones. The assessment is a structured output with a flag, a reason, a category and a priority from 1 to 10, and the probability Claude assigns to "flagged" becomes a score that can be thresholded.

At a 1% false positive rate on production traffic with Claude Fable 5, the paper reports recall of 78% overall and 94% for high-priority conversations, with an area under the curve of 0.986. Broken out by priority, recall was 71% for low (49 conversations), 79% for medium (67) and 94% for high (17).

Anthropic estimates the technique adds under 2% to inference compute, because the conversation is already cached and each assessment generates about 10 tokens on average.

Reading the numbers like an editor

Three habits make these claims easier to read.

Check the sample size. The 94% for high-priority cases rests on 17 conversations. Missing one of 17 would move it by about six points. The paper gives the counts, which is to its credit, but the headline figure is the one that travels.

Do the false positive arithmetic. A 1% rate sounds tiny. On one million harmless conversations it means about 10,000 wrongly flagged ones. That is our illustration, not a figure from the paper, but the paper makes the same point: "Even a 1% false-positive rate produces many benign flags", and so uses of the tool "rely on the priority score and on repeated flags, not on any single assessment".

Ask who set the answer key. The monitor learned from a teacher, a prompted classifier based on Claude Mythos Preview, and is evaluated against that same teacher. The authors say blind spots in the teacher pass through undetected. A monitor can look excellent while sharing its teacher's mistakes.

What the paper admits

The limitations section is blunt. "It does not catch every harmful conversation, and it is not robust enough to stop a dedicated attacker," the authors write. In a test of 24 offensive cyber tasks with production classifiers off, attack success was 79% with no safeguards, 54% with self-assessment acting as a blocker and 25% with a prompted classifier. As an offline monitor, self-assessment flagged 21 of the 24 attack trajectories at least once, so it works better as a tripwire for review than as a gate.

On the harder question, whether a model can monitor itself, the authors conclude that self-assessment "is not yet a reliable monitor of Claude's own conduct". A deceptive model could hide a misaligned action in its assessment as well, and a shared judgment can make the same error in both action and assessment.

How labs combine monitors

No lab relies on one monitor. In a layered setup, cheap checks run on everything, stronger reviewers run on what the cheap checks flag, and humans handle the top of the queue. OpenAI's recently described design, as reported by Unite.AI, sends a random 5% of unflagged traffic from its first layer to review, specifically to measure what the cheap check misses. That sampling is the honest way to estimate recall in production, because it tests the monitor on traffic it did not flag.

Our explainer on AI control and untrusted monitors covers why labs assume the model might be working against them, and why monitors must be tested by people who did not build them.

How to use this

When a lab reports a monitor, ask five questions. What recall, at what false positive rate? On how many real cases? Graded against whose labels? What does the monitor do when it fires: block, flag or only log? And what did an outside team find when it tried to evade it? Anthropic answers the first three in its paper. The last two are where independent red teams, such as METR and Redwood Research, add the most.

Our position: a 78% catch rate at a 1% false alarm rate is a useful second line of defense and a poor first one. Read it as a measurement of how much a tripwire sees, not as proof that the harm it misses is small.