Developing nuclear safeguards for AI

Anthropic and the US National Nuclear Security Administration co-developed a classifier that flags concerning nuclear-weapons conversations; on a synthetic test set it caught 94.8% of harmful prompts with no false positives (96.2% accuracy), and in deployment it flagged some benign current-events conversations that a second monitoring step cleared.

Anthropic (Frontier Red Team), with the NNSA and DOE national laboratories · 2025-08-21 · original

This post reports how Anthropic and the US Department of Energy's National Nuclear Security Administration (NNSA) moved from testing Claude for nuclear proliferation risk to building a monitor for it. Information about nuclear weapons is classified, which makes it hard for a private company to evaluate such risk alone; the question the post addresses is how a government agency and a company can build and validate a safeguard when neither can show the other its sensitive data. It was published on 21 August 2025 on the Frontier Red Team's blog, red.anthropic.com, and now appears on anthropic.com under the Frontier Red Team label with the same text. A shorter news post of the same date ("Developing nuclear safeguards for AI through public-private partnership") repeats the opening and links to the full version. Neither names individual authors. The post is a short announcement: it gives one test result, a deployment anecdote and a description of process, and no technical report accompanies it.

What was built

The post says Anthropic partnered with the NNSA "last April" to assess its models for nuclear proliferation risk (introduction), and links an Axios report of 14 November 2024 in which Anthropic said it had worked with the NNSA "since April", so the April meant is April 2024, although a reader in August 2025 could take "last April" as April 2025 (the year is the note's reading of the linked report). The Claude Opus 4 system card instead says Anthropic has "maintained a formal partnership" with the NNSA "since February 2024" (Section 7.2.2); the sources do not say whether the two dates mark different stages. After about a year of NNSA staff red teaming Claude models in a secure environment, the two sides began co-developing mitigations (section "What we built").

  • Risk indicators. Informed by its red teaming, the NNSA gave Anthropic a curated list of nuclear risk indicators meant to separate potentially concerning conversations about weapons development from benign discussion of nuclear energy, medicine or policy. The list was prepared "at a classification level such that it could be shared" with Anthropic's team. The post does not publish the list.
  • Classifier. Anthropic's Policy and Safeguards teams turned the list into a classifier, a system that labels conversations, which runs on Claude traffic in real time. The post does not say what model the classifier is built on or how it was trained.
  • Validation by synthetic data. Neither side could share its sensitive material: the NNSA's classified information and Anthropic's user data. Anthropic therefore used example prompts from the NNSA to generate "hundreds" of synthetic test prompts, some concerning and some benign, ran them through the classifier, and shared the scores with the NNSA, which checked that they matched the expected labels. The cycle was repeated to improve precision (section "What we built"; Figure 1, a five-step diagram from risk identification to performance validation).

What it found

  • Synthetic test. In "preliminary testing with synthetic data", the classifier detected 94.8% of nuclear-weapons queries and produced no false positives; overall, 96.2% of its labels were correct (section "Why does this matter?"; Figure 2, a confusion matrix printing true negatives 100%, false positives 0%, false negatives 5.2%, true positives 94.8%, accuracy 96.2%, precision 100%, recall 94.8%). The post does not give the size of the test set beyond "hundreds" of prompts, the share that was harmful, or any interval. From the printed rates, the accuracy implies that roughly 73% of the test items were harmful (this arithmetic is the note's, not the post's). The headline "96% accuracy" in the introduction is this figure rounded. The post reads the absence of false positives as "suggesting this system would not flag legitimate educational, medical, or research discussions" and says the classifier "appears to strike the right balance"; its own deployment observations (below) show benign conversations flagged.
  • Deployment. The classifier was deployed "as an experimental addition" to Anthropic's safeguards, monitoring "a percentage of Claude traffic" (section "What's next?"). The post says real-world performance "confirmed that the classifier works effectively", but reports no deployment rates. It gives two observations instead.
  • Deployment false positives. During a period of renewed attention to nuclear weapons following events in the Middle East, the classifier wrongly flagged some conversations that were only about those events. When those flags went through hierarchical summarization, a monitoring step that reviews several flagged conversations together, they were recognized as benign current-events discussion.
  • Deployment true positives. Anthropic red teamers who did not know the classifier was deployed sent deliberately concerning prompts during routine testing, and the classifier flagged them.

From these the post draws two lessons: real conversations fall into gray areas that synthetic data does not capture, and combining several safety tools makes the overall system more precise.

Sharing the approach

Anthropic said it would share its approach with the Frontier Model Forum, the industry body of frontier AI developers, "in hopes that this partnership can serve as a blueprint" for other developers working with the NNSA (introduction). A footnote explains that this sharing is possible because Forum members (Amazon, Anthropic, Google, Meta, Microsoft and OpenAI) signed an information-sharing agreement on threats, vulnerabilities and capability advances (footnote 1).

Instructions and texts involved

The post contains no prompts or principles that it prints. The governing text is the NNSA's list of nuclear risk indicators, which defines what the classifier should treat as concerning. It is a written specification, drafted by a government agency from its red-teaming findings and turned by Anthropic into a classifier, but it is not published, and the post says nothing about its wording or how it was converted. In this respect it parallels the written constitution of restricted and permitted content used to train Constitutional Classifiers (the comparison is the note's; the post does not say whether the same method was used).

Limitations

  • Synthetic, small and unsized test. The 96.2% figure comes from model-generated prompts derived from NNSA examples, counted only as "hundreds", with the class balance unstated and no interval. The post itself calls it preliminary.
  • Zero false positives on synthetic data, not on traffic. The post's own deployment anecdote shows false positives on real conversations; no false-positive rate on real traffic is reported.
  • Unverifiable inputs. The indicator list, the classifier's design and the deployment data are not public, so the result cannot be checked outside the partnership.
  • Anecdotal deployment evidence. The true positives in deployment were Anthropic's own red teamers, not real misuse attempts.
  • Grading. The NNSA validated that scores matched expected labels; the post does not say who wrote the labels or how disagreements were resolved.

Research questions

  1. What are the classifier's error rates on real traffic? Precision and recall on a labeled sample of deployed nuclear-related conversations would show how far the synthetic-test figures carry over.
  2. How much does hierarchical summarization reduce false positives? Comparing flag rates before and after the summarization step, by topic and period, would quantify the layered-system claim.
  3. How well does a synthetic test set built from a few agency examples cover real misuse? Testing the classifier on prompts written independently by other red teamers would show whether the test set shares blind spots with the classifier.
  4. Was the approach adopted by other developers? Public statements by Frontier Model Forum members or the NNSA about similar classifiers would show whether the "blueprint" was used.
  5. How does the indicator list translate into classifier behavior? A description, at an unclassified level, of how indicators were turned into training data would show how much the written specification determines what is flagged.

How it connects

References