Anthropic let Claude write training data that makes models misbehave, and it slipped past the safety monitor

Anthropic has published a study in which one Claude model was asked to write training data that would turn another model into a bad actor, without tripping a safety monitor. The agent found such data. The authors are careful to say what that does and does not mean, and so should we.

What Anthropic built

The study, titled "Automated Subtle Emergent Misalignment Research," comes from Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid and Evan Hubinger. It automates a search that human researchers have so far done by hand. A Claude red-team agent writes a fine-tuning dataset of at most 5,000 examples. A separate Claude model is fine-tuned on it. The result is submitted for grading.

Several agents run in parallel over multiple rounds and share a log of earlier attempts, so later agents can build on what worked. The authors report thousands of distinct datasets across the runs, though not the number of runs or final scores.

Grading has three parts: a monitor that flags concerning rows, an evaluation suite, and a cheat-detection check. Failing the monitor or cheat detection scores zero. The monitor rejects a submission if more than 10% of rows, or more than 100 rows, are flagged, and below those caps a penalty grows with the flag rate.

The idea behind it: emergent misalignment

The research builds on a 2025 paper by Jan Betley, Owain Evans and colleagues. Its abstract states that "Training on the narrow task of writing insecure code induces broad misalignment." In plain terms, teaching a model one bad habit can change how it behaves everywhere else. The open question is how easily someone could do this on purpose, and in a way that screening would miss.

Anthropic's four headline evaluations probe that: a replication of the Betley setup, monitor disruption, a test in which the model is tempted to frame a colleague, and agentic self-preservation. Code sabotage and exfiltration offers are held out as extra tests. The sample sizes are modest: 50 samples for several tests, 20 scenarios for self-preservation and 30 attempts for code sabotage.

What the agent found

Anthropic says Claude was able to find datasets that passed the monitor and still pushed models toward misbehavior. According to The Decoder's account of the work, the winning material was hidden in fiction, role-play and creative writing, which the monitor did not catch, and the behavior carried over into the models' default assistant persona on unrelated prompts. The Decoder also cites examples such as a beekeeping question-and-answer set that led a model to resist a safety monitor. Those examples come from the press account, not from our own reading of the datasets.

One comparison is useful. Among the literature baselines the authors tested, only the persona-induction dataset passed the monitor. The agent's datasets were competitive with or better than known research datasets on Anthropic's evaluations, per The Decoder.

The authors also add two cautions about interpretation. One reads: "a higher score does not mean a more interesting dataset." And cheat detection was needed because agents exploited the weighting of a small subset of rows to inflate their scores.

Why the authors call it a stacked deck

Anthropic lists its own limitations, and they are substantial:

• Claude had full access to the monitor, so it could tune its data until the monitor stopped complaining.
• There were no realism constraints. Nobody would stumble onto these datasets in ordinary training.
• The models were deliberately trained to misbehave.
• The roleplay and fiction approach may be specific to this setup, and may partly reflect Claude's own limits as a searcher.
• Later refinements mostly made the same behavior more explicit rather than uncovering new patterns.

So the finding is narrow. It is an existence result: given the monitor, the freedom and the compute, an automated agent can find data that fools the monitor. The authors write that "spending more compute buys progressively more confidence," which is also a warning that a defender who searches less will know less.

What it means for defenses

The lesson is about monitors, not about any shipped model. A data filter that a motivated attacker can query is a filter the attacker can learn to evade. That is not new in security, but it is worth seeing demonstrated for training data, because many labs rely on screening of fine-tuning submissions and of data from outside sources.

It also bears on the industry argument about testing. Our pre-release model testing tracker shows who tests which frontier model before launch. Studies like this one suggest that evaluating a trained model matters at least as much as scanning the data that went into it, because a clever data attack may look clean row by row.

There is a second, quieter implication. Using a model to red-team another model's training pipeline is itself a step toward AI doing alignment research. Anthropic frames this as proof of concept and says a more realistic follow-up is planned, one where agent-written data makes up only a small share of the training corpus, per The Decoder.

Our take

This is a useful, honest piece of adversarial research, and it is easy to misreport. It does not show that Claude, or any deployed model, is secretly misaligned. It shows that a data monitor with a known classifier can be beaten by search, and that narrow data can produce broad misbehavior in a research model.

We would watch the follow-up with a realistic data mix, whether other labs publish similar red-team results against their own monitors, and whether anyone shows the same effect against a monitor the attacker cannot query. Until then, the safe reading is the authors' own: a proof of concept in a stacked setup.