Introducing Anthropic's Safeguards Research Team

A short February 2025 post announcing Anthropic's Safeguards Research Team, formed after the Constitutional Classifiers release and led by Mrinank Sharma, with its planned work in five areas (defenses against misuse and misalignment, finding vulnerabilities, monitoring, response, and safety cases) and its working ties to Anthropic's Safeguards organization, the Anthropic Fellows Program and MATS.

Anthropic (Alignment Science Blog) · 2025-02-17 · original

The Safeguards Research Team is an Anthropic research team working on defenses against misuse of models and against model misalignment. This short Alignment Science Blog post, of under 400 words, announced it "following the release of Constitutional Classifiers", a paper of early February 2025 whose first author, Mrinank Sharma, leads the new team (Constitutional Classifiers). The blog's index lists the post under February 2025; the first Wayback Machine capture is from 17 February 2025, which this note gives as the latest possible date. The post reports no results. It states who is on the team, how the team relates to the rest of Anthropic, and what it plans to work on. This note follows the text retrieved on 13 November 2026; the Wayback Machine's captures from February and March 2025 have two different contents, and neither could be retrieved for comparison.

Who is on the team

The post says the team "is currently led by Mrinank Sharma" and names nine other members: Erik Jones, Meg Tong, Jerry Wei, Euan Ong, Alwin Peng, Ted Sumers, Taesung Lee, Giulio Zhou and Scott Goodfriend. "Currently" refers to the post's date; the post gives no later roster.

How the team relates to the rest of Anthropic

The team is a research group that works "in close collaboration" with Anthropic's broader Safeguards organization, which the post describes as responsible for deployment infrastructure, safety mechanisms and user policies. The post's stated reasons for the arrangement are that it lets the team inform threat models, watch for early signs of model misalignment, and get "rapid feedback" on whether its techniques work in production. The post also says the team works closely with the Anthropic Fellows Program and will run some projects through MATS, an outside research-training program.

The five planned areas

The post lists planned work, each with examples of earlier work it links:

  • Defense techniques for misuse and misalignment: classifiers such as Constitutional Classifiers and other classifier-based approaches (arXiv:2412.02159), "potentially using model internals" (a latent adversarial training paper, arXiv:2407.15549).
  • Vulnerability identification: Many-shot Jailbreaking, best-of-N jailbreaking (arXiv:2412.03556) and work on image jailbreaks (arXiv:2407.15211), plus automated red teaming.
  • Monitoring: efficient monitors, threat modeling informed by deployment, and AI-assisted summarization to surface misuse not foreseen in advance, with Clio and anomaly detection as examples.
  • Response: Rapid Response, preventing a catastrophe after catching a few examples (a LessWrong post on "catching AIs red-handed"), and protocols for low-stakes control (arXiv:2411.17693). The post frames the area as a question: "If we detect harm, how can we respond quickly to mitigate it?"
  • Safety cases: documented arguments for a system's safety, "particularly for model misuse".

Later work in the collection

The post makes plans; it does not report later work. This note connects it to later items in the collection on the same problems, as the note's own reading: Monitoring computer use via hierarchical summarization (February 2025, first author Theodore Sumers, on the monitoring area's AI-assisted summarization), Constitutional Classifiers++ (January 2026, a successor to the classifiers the post names) and Poisoning Fine-tuning Datasets of Constitutional Classifiers (April 2026, a Fellows study whose authors include Jerry Wei, named above). Neither this post nor those items' bylines, which list authors and not teams, say which of them the Safeguards Research Team produced.

Limitations and open questions

The post is a team announcement and recruiting page; it lists intentions without timelines or measures of success. It does not say how the team's research is weighed against product needs in the Safeguards organization, or what the team will publish. The "safety cases" area is named in one sentence with no linked example.

Research questions

  1. Which of the five planned areas led to published work? Mapping the team's later papers to the areas would show where effort went.
  2. How did closeness to the production Safeguards organization affect what was published? Comparing the share of work evaluated on production systems with that of other alignment teams would show the effect the post expects.
  3. Did the team publish a misuse safety case? A search of later Anthropic safety reports and system cards for a misuse safety case credited to the team would answer it.
  4. How did the team's roster change? Later bylines and announcements would show whether the named members and lead stayed with the team.

How it connects

References