What technical work done now would help a developer keep much more capable future systems safe? This post, published on the Alignment Science Blog on 10 January 2025, is Anthropic's Alignment Science team's answer in the form of a list of open problems. Its byline credits "The Alignment Science team, with organizing from Sam Marks and Peter Hase," and its acknowledgements thank Michael Lepori and Atticus Geiger for feedback on a draft. Sam Marks crossposted it to LessWrong and the Alignment Forum the same day. The post is about 5,200 words and reports no experiments or measurements. Its claims are proposals and judgments about which problems matter, supported by citations to other work. This note follows the live page, read on 22 November 2026.
What the post is, in its own terms
The team says it asked its members "to write descriptions of open problems or research directions they thought were important" and grouped the answers (introduction). Two disclaimers set the scope:
- It is "not a comprehensive list of technical alignment research directions"; the post calls it "a tasting menu."
- It is "not a list of directions that we are actively working on, coordinating work in, or providing research support for." The team does research in some of the areas, but many are directions it "would like to see progress in, but don't have the capacity to invest in ourselves."
The motivating risk is stated at the start: catastrophes "caused by future advanced AI systems, such as mass loss of life or permanent loss of human control." The audience is outside researchers who want to reduce that risk and ask what work "AI developers will find useful for ensuring the safety of their future systems."
The six groups
The post has six top-level sections and fourteen subsections. Every subsection, and every section except Miscellaneous, opens with a one-line question. Headings below are the page's own.
Evaluating capabilities
Benchmarks, including PhD-level ones, "saturate quickly," while real-world impact "often lags behind" benchmark scores. The post asks for evaluations that track real-world impact, of capabilities such as novel research, tool use and open-ended autonomous tasks, "alongside human baselines." It says evaluations of chemical, biological, radiological and nuclear (CBRN) capabilities "should be conducted in secured labs in coordination with governments." It points to Anthropic's 2024 third-party evaluations initiative, noting that "the deadline for that initiative has passed."
Evaluating alignment
Current alignment measurements, the post says, focus on "relatively surface-level properties": pleasant conversation, refusing harmful queries, avoiding toxic text. These "suffice for assuring the safety of current AI chatbots," but not for future systems in unanticipated, high-stakes settings. The questions it wants answered include whether a model has goals or preferences, whether it fakes being more aligned than it is or hides a capability, and when it would knowingly mislead a human. It calls these "extremely subtle" and often "vague or ill-defined." Three subsections follow:
- Understanding model cognition. Checking whether outputs come about "for the right reason": whether a model forms plans, what it knows about its situation, whether it knows things it does not report. The post leaves mechanistic interpretability to the interpretability team's blog and names two other approaches: externalized reasoning (chain of thought) and introspection, meaning "Training models to directly verbalize the content of their hidden states," with a proposed test of whether a model's reported plan in a board game predicts its later moves.
- Understanding how a model's persona affects its behavior and how it generalizes out-of-distribution. One hypothesis the post offers is that leading models generalize benignly because they adopt a helpful-assistant persona. It cites its own alignment-faking work, in which "in certain settings, only some models would fake alignment," and proposes building models of similar capability with different personas, by few-shot prompting or synthetic-data training, to see how persona changes behavior.
- Chain-of-thought faithfulness. Called "an important but underexplored area of research." Questions include how faithfulness varies by task (math against social reasoning), whether a performance gain from reasoning implies faithful reasoning, and whether models reason more or less faithfully when told their reasoning is monitored.
AI control
The section sets out three kinds of safety assurance. The first, lack of dangerous capability, is the post's own reason for believing current systems are not catastrophically risky: "the main reason we believe AI systems don't pose catastrophic risks is that they lack many of the capabilities necessary for causing catastrophic harm." The second is sufficient alignment, once alignment can be evaluated. The third, AI control, is deploying a model with safeguards it cannot circumvent "even if they tried." The post says control is appropriate for models capable enough to cause a catastrophe "if deployed with no safeguards" but not capable enough to evade a careful developer's safeguards. Its subsections:
- Behavioral monitoring. A monitor model screens an untrusted actor's inputs and outputs, as in trusted monitoring (a weaker trusted model, such as GPT-3.5 checking GPT-4's code in Greenblatt et al., 2023) and untrusted monitoring (another instance of the actor). The core problem is that "monitor failures might be systematically concentrated in the same episodes where the actor behaves maliciously," through jailbreaking or collusion. Proposed work includes paraphrasing the actor's output before the monitor sees it, synthetic training data for monitors, estimating rates of very rare catastrophic outcomes, and monitoring actors that have memory or can pass messages between instances.
- Activation monitoring. Probes on a model's internal activations, which the post says correspond to the monitored model's reasoning and "are difficult for the model to control." The stated risk is that such monitors "are not robust to distributional shifts," and the post names "one particularly important distribution shift": from fixed synthetic datasets to data the model generates itself. The post says there are "signs of life," linking a LessWrong post on probes and the circuit-breakers paper (Zou et al., 2024), and that more research is needed before production use.
- Anomaly detection. Unsupervised detection of unusual computation, to flag rare inputs such as jailbreaks that a supervised monitor misses. A more ambitious version would flag inputs where a strong model's behavior surprises a weak one; the post calls this "as far as we know, an unexplored problem setting."
Scalable oversight
Scalable oversight is "the problem of designing oversight mechanisms that scale with the intelligence of the systems we aim to oversee." The post names three difficulties at the frontier of human knowledge: noisy oversight on hard problems, systematic errors, and cost. Four subsections follow.
- Improving oversight despite systematic, exploitable errors in the oversight signal. The post's framing is a proxy reward R that errs systematically against an inaccessible ground truth R*. If the model can model the difference, training on R may teach it to exploit the errors, giving answers that look correct to overseers but are not its best guess. The post proposes a testbed, described below under "The prompts and instructions in the proposals."
- Recursive oversight. Task decomposition, debate and prover-verifier games, which build an improved reward R′ from R with AI help, ideally repeatedly. Of the adversarial methods the post says: "The hope is that the only equilibrium in these adversarial games is for the model to perform as well as they can."
- Weak-to-strong and easy-to-hard generalization. Training on imperfect supervision as it is and hoping for favorable generalization: a strong model trained by a weak overseer (GPT-4 supervised by GPT-2 in Burns et al., 2023), or a model trained only on easy tasks with reliable labels and tested on hard ones. The post's citation "Furi et al., 2023" links to arXiv:2306.09983, whose first author is Lukas Fluri.
- Honesty. The hope that honest responses have "a common, identifiable structure," so that a model's own knowledge could show when it answers honestly without the overseer knowing the right answer. The post cites work on truth representations and asks when such techniques work, "especially with a focus on situations where models have learned to give non-truthful answers."
Adversarial robustness
The post asks for better maps of future attack surfaces (it names pretraining-data poisoning and malicious content on websites that hijacks agents), defenses that interfere little with benign use, and better measures of how reliable defenses are. Two subsections:
- Realistic and differential benchmarks for jailbreaks. The post says that "we ultimately don't care if the model responds to all questions it would refuse by default"; what matters is whether answers enable real misuse. The post asks for benchmarks of differential impact (what an attacker can do with the model that they could not do with tools such as search engines), realistic harmful tasks, universality across inputs and models, and how often real people find jailbreaks.
- Adaptive defenses. Defenses that change with the attacker: monitoring sets of queries rather than single ones, and rapid response, patching a model soon after an attack is found.
Miscellaneous
- Unlearning dangerous information and capabilities. The post says that "the problem remains largely unsolved," since apparently unlearned information can still be extracted. It encourages methods whose models "behave near-identically to models that were never trained on unlearned data," even after fine-tuning meant to recover it.
- Learned governance for multi-agent alignment. Many interacting instances can fail together "even if each individual instance is aligned to human values." The post lists four failure modes: small externalities that add up, problems passed along without reaching anyone who can fix them, diffusion of responsibility, and information not shared. It suggests that governance "might be feasible to learn" from observations, and says the direction "appears ripe for theory-driven approaches" drawing on game theory, as well as empirical work.
The prompts and instructions in the proposals
The post is an agenda and contains no trained system. Three of its proposals use written instructions as an experimental lever:
- The oversight testbed. Start from a student model M and a ground-truth signal R*; add systematic errors to make a training signal R; "Ensure that M understands the systematic errors made by R," for example "by providing M a system prompt that explains some of R's errors"; train M on R, optionally with a scalable-oversight method; and score M against R*. In the post's example, a prompt supplies the model's knowledge of its overseer's mistakes.
- Personas. A suite of models with different personas could be built "with few-shot prompting or training on synthetic data."
- Telling a model it is monitored. One faithfulness question is whether models "produce more or less faithful explanations after being told that their CoT is being monitored or evaluated for faithfulness."
The post gives no wording for any of these prompts.
Limitations
- No evidence of its own. Every claim about what works or fails is a judgment or a citation. The post gives no ranking of directions, no criteria for including one, and no count of how many team members contributed.
- Selection. The list is the team's members' own choices, and the post says it is incomplete. Directions the team works on and directions it hopes others will take up are mixed, and the post does not say which are which.
- Stake. The post asks outside researchers for work that developers, Anthropic among them, "will find useful," in many areas the team says it lacks capacity to pursue. Its view that current systems do not pose catastrophic risks mainly because they lack the necessary capabilities is the developer's own assessment.
- Cited models. Its examples of model behavior come from papers of 2023 and 2024 (GPT-4 supervised by GPT-2, GPT-3.5 monitoring GPT-4, and the Claude models of the sycophancy and alignment-faking studies). It makes no claim about which later models the problems apply to.
- Peer review. It is a blog post. The one comment on the LessWrong crosspost (10 January 2025) asks how much Anthropic's staff are sped up by their own models; it does not discuss the directions.
Research questions
- Which of the listed directions has Anthropic itself since published on? Mapping later Anthropic papers to the fourteen subsections would show which directions moved from "would like to see" to the developer's own work, and which stayed open.
- Does the proposed oversight testbed work as described? An experiment that gives a student model a system prompt describing a reward signal's planted errors, trains it on that signal, and scores it against ground truth would show whether the model learns to exploit the errors it was told about, and whether a scalable-oversight method reduces this.
- Do models reason more or less faithfully when told their reasoning is monitored? Faithfulness measured with and without such a statement in the prompt, on the same tasks, would answer the post's question directly.
- Do activation monitors trained on synthetic examples catch a model's own harmful outputs? Detection rates on held-out on-policy generations against rates on the synthetic training distribution would measure the shift the post names as the main risk.
- Can anomaly detection flag inputs where a strong model surprises a weak one? A benchmark of inputs labeled by whether two models' behavior diverges, with detection rates for unsupervised methods, would test the problem the post calls unexplored.
How it connects
The post cites several items that have notes here, and later notes in the collection take up many of its directions. Except where the post cites an item, the connections below are this note's.
- Cited by the post. Towards Understanding Sycophancy in Language Models (October 2023) is its example of a model's tendencies; Alignment faking in large language models (December 2024) is its example of models that fake alignment, and only some of them; Measuring Faithfulness in Chain-of-Thought Reasoning and Question Decomposition Improves the Faithfulness of Model-Generated Reasoning (July 2023) are cited as ways to detect and to improve faithfulness; Rapid Response (November 2024) is the rapid-response link.
- Evaluating alignment. Auditing language models for hidden objectives (March 2025) tests whether auditors can find a goal a model was trained to hide.
- Persona. Why Do Some Language Models Fake Alignment While Others Don't? (June 2025) runs the alignment-faking setting on 25 models; Persona Vectors (August 2025) monitors and steers traits through activation directions.
- Chain of thought and introspection. Reasoning Models Don't Always Say What They Think (April 2025) measures how often reasoning models mention hints they used; Emergent Introspective Awareness in Large Language Models (October 2025) and Introspection Adapters (April 2026) test models' reports about their own states.
- Control and monitoring. SHADE-Arena (June 2025) evaluates sabotage and monitoring in agents; Cost-Effective Constitutional Classifiers via Representation Re-use (June 2025) tests probes on activations as classifiers; Simple probes can catch sleeper agents (April 2024) is earlier Anthropic work on activation probes. A "diff" tool for AI (March 2026) compares two models' features, a different route to the post's interest in detecting large differences between two models.
- Scalable oversight and honesty. Measuring Progress on Scalable Oversight (November 2022) is Anthropic's earlier experimental design; Unsupervised Elicitation (June 2025) trains on a model's own consistent labels without gold labels; Evaluating honesty and lie detection techniques (November 2025) compares honesty and lie-detection methods on models that lie.
- Robustness and unlearning. Constitutional Classifiers (February 2025) is a misuse defense tested against human red-teamers; A small number of samples can poison LLMs of any size (October 2025) studies pretraining poisoning; Beyond Data Filtering (December 2025) and An off switch for dual-use knowledge (July 2026) take capability removal into pretraining.
- Multi-agent systems. Patterns and problems in emerging multiagent systems (August 2026) and Worse Together (November 2026) report failures of groups of agents.
- Programs. Introducing the Anthropic Fellows Program (November 2024) and its 2026 call list the research areas Anthropic mentors outside researchers in.
References
- Anthropic Alignment Science team, with organizing from S. Marks and P. Hase. (2025, January 10). Recommendations for Technical AI Safety Research Directions. Alignment Science Blog. https://alignment.anthropic.com/2025/recommended-directions/index.html (read 22 November 2026). record
- Marks, S. (2025, January 10). Recommendations for Technical AI Safety Research Directions (crosspost). LessWrong. https://www.lesswrong.com/posts/tG9LGHLzQezH3pvMs/recommendations-for-technical-ai-safety-research-directions
- Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. (2023). AI Control: Improving Safety Despite Intentional Subversion. arXiv:2312.06942. record · 4 pages
- Burns, C., Izmailov, P., Kirchner, J. H., et al. (2023). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. arXiv:2312.09390. record · 2 pages
- Fluri, L., Paleka, D., and Tramèr, F. (2023). Evaluating Superhuman Models with Consistency Checks. arXiv:2306.09983. (Cited in the post as "Furi et al., 2023".) record
- Zou, A., Phan, L., Wang, J., et al. (2024). Improving Alignment and Robustness with Circuit Breakers. arXiv:2406.04313. record