Trustworthy agents in practice

AI agents can carry out multi-step tasks across files, applications and the web with less human oversight than a chatbot, which makes them both more useful and more open to misreading a user's intent or being hijacked by malicious content. This post, published on 9 April 2026 under Anthropic's Policy label, explains how Anthropic applies the framework for trustworthy agents it published in August 2025 and where it thinks industry, standards bodies and governments should build shared infrastructure. The post links to a longer document, Anthropic's response of 9 March 2026 to a Request for Information from the Center for AI Standards and Innovation (CAISI) at the US National Institute of Standards and Technology (NIST) on the security of AI agents (Docket NIST-2025-0035). This note covers the post first and then summarizes the submission, naming the source for each point.

The framework and the definition of an agent

The August 2025 framework, as the post summarizes it, rests on five principles: keeping humans in control, aligning with human values, securing agents' interactions, maintaining transparency, and protecting privacy (introduction). The post discusses three of them, human control, alignment with user expectations and security, and says transparency and privacy run through each.

The post defines an agent, following Anthropic's earlier Building effective agents, as an AI model that directs its own processes and tool use, deciding how to achieve what users want rather than following a fixed script (section "How agents work"). An agent works in a loop: it plans, acts, observes the result, adjusts and repeats until the task is done or it needs human input. The post's example is Claude in Claude Cowork submitting expense receipts, noticing that a hotel charge was flagged for exceeding a nightly cap it does not know, and asking whether to fetch the company's expense policy before trying again.

Four layers

The post describes an agent as built from four components, each "both a source of capability and a potential point of oversight":

• The model, whose knowledge and behavior come from training.
• The harness, the instructions and guardrails the model operates under; in the receipts example, a rule to flag anything over a hundred dollars or never to submit without user confirmation.
• Tools, the services and applications the model can use, such as email, calendar or expense software.
• The environment, the product the agent runs in and the files, websites and systems it can reach. The same agent on a corporate laptop and on a personal phone has different data access and different stakes.

The post argues that most AI policy discussion focuses on the model, but that an agent's behavior depends on all four layers: a well-trained model can still be exploited through a poorly configured harness, an overly permissive tool or an exposed environment.

The principles in practice

Human control

In Claude.ai and Claude Desktop, users choose which tools to enable and set permissions for each action (always allow, needs approval, or block), so that reading a calendar can be automatic while sending an invitation needs approval (section "Designing for human control"). The post notes that for tasks with dozens of actions, repeated approval prompts become friction that users sometimes tune out. Claude Code's Plan Mode addresses this by showing the whole intended plan up front for the user to review, edit and approve, while still allowing intervention during execution. The post describes this as moving oversight from the individual step to the overall strategy, "which we find tends to be where users most want to exercise judgment." It also flags subagents, other Claude instances working in parallel on parts of a task, as an open problem for oversight, since the work is no longer a single visible thread.

Helping agents understand their goals

The post calls knowing when to stop and ask "one of the harder unsolved problems in agent development" (section "Helping agents understand their goals"). An agent must tell gaps it can resolve itself from questions of preference or intent that only the user can settle, and pausing too often costs autonomy while pausing too rarely risks misreading intent. Anthropic says it addresses this in training in two ways: by building training scenarios that place Claude in ambiguous situations and reinforcing the choice to pause rather than assume, and through Claude's constitution, which the post says "directly shapes how our models are trained" and favors "raising concerns, seeking clarification, or declining to proceed" over acting on assumptions.

As evidence of the effect, the post cites Anthropic's research on agent use: on complex tasks, users interrupt Claude only slightly more often than on simple ones, but Claude's own rate of checking in roughly doubles.

Defending against prompt injection

Prompt injection is the post's name for malicious instructions hidden in content an agent processes, such as an email telling the agent to forward messages to an attacker (section "Defending against attacks"). The post says no single defense is enough: the more open the environment, the more entry points, and the more tools, the more an attacker can do. Anthropic trains the model to recognize injection patterns, monitors production traffic to block real attacks, and uses external red-teamers. Because these are still not a guarantee, it advises customers to think carefully about which tools, data, permissions and environments they give an agent.

What the post asks of others

The section "What the broader ecosystem can do" proposes three contributions from outside any one company:

• Benchmarks. There is no rigorous, standardized way to compare agents on prompt-injection resistance or on how reliably they surface uncertainty; companies test with their own unverified methods. The post suggests standards bodies such as NIST maintain shared benchmarks and encourage third-party evaluation.
• Evidence sharing. Anthropic hopes publishing evidence on how agents are used and where they struggle becomes common practice.
• Open standards. Anthropic created the Model Context Protocol (MCP) as an open standard for connecting models to data and tools and has donated it to the Linux Foundation's Agentic AI Foundation. The post argues that open protocols let security be designed into infrastructure once and keep competition focused on the quality and safety of agents.

The companion NIST submission

The submission, dated 9 March 2026, answers selected questions from the RFI in more technical detail. Points that go beyond the post:

• A failure category without a name. The submission argues that NIST's security frameworks address external attackers and deliberate misuse but not "a well-functioning, non-compromised agent" that stays within its permissions and still takes harmful actions (response to Question 1(a)). Its examples include an agent that reads "delete all emails from the last month and all emails from a specific person" as a union rather than an intersection, and one that infers salaries from documents it was allowed to read. It traces the gap through several NIST documents and asks NIST to develop shared vocabulary for such failures, while acknowledging that whether a behavior counts as a failure often depends on context no definition can fix in advance.
• Threat vectors. It names indirect prompt injection, persistent memory poisoning (corrupted information saved into an agent's memory and acted on later) and tool supply-chain compromise, noting that these are drawn from Anthropic's research and red-teaming rather than observed incidents.
• The four layers as a security argument. Most security evaluation focuses on the model, but what an agent can do when something goes wrong is set by the harness and execution environment: "The failure is identical. The consequences are not." (response to Question 2(a)). Later it proposes asking not whether a model can be compromised but "what is the scope of damage if it is?" (response to Question 4(a)).
• Model behaviors and their maturity. The submission names surfacing uncertainty, presenting a plan before execution, and flagging irreversible actions. It reports that on the most complex tasks Claude asks for clarification on 16.4% of turns, more than twice the rate on minimal-complexity tasks, and adds: "We do not want to overstate this. Claude may not always stop at the right moments, and product design shapes the behavior as well." It cites Constitutional AI and Constitutional Classifiers as the training and classifier methods behind these behaviors, and Anthropic's sabotage evaluations and SHADE-Arena as tests showing "meaningful but incomplete coverage."
• Oversight shifting with experience. Citing the same agent-autonomy research, the submission reports that experienced Claude Code users auto-approve roughly twice as often as new users and also interrupt more often, which it summarizes as "Oversight has not decreased, it has moved" (response to Question 5(c)). It argues that a theoretical model would have predicted the opposite, as a case for empirical research.
• Policy positions. It favors measurement, evaluation and information sharing over prescriptive mandates, open industry-led standards bodies, and agent identity standards on the web that do not let intermediaries block or charge identified agents by default.

The written principles

The post cites a written document as one of two training mechanisms for a specific agent behavior. It states that Claude's constitution "directly shapes how our models are trained" and that the constitution favors "raising concerns, seeking clarification, or declining to proceed" over acting on assumptions (section "Helping agents understand their goals"). In the constitution itself, the quoted phrase appears in the section "Having broadly good values and judgment," in a passage on when Claude should exercise independent judgment (its example is discovering evidence of an operator's fraud during an agentic task), where these options are described as "generally preferable to unilateral intervention." In the constitution the comparison is with drastic unilateral action; the post presents the same options as preferred over acting on assumptions, in the context of when an agent should pause and ask. The post presents the constitution alongside, not instead of, targeted training scenarios, and does not separate their contributions.

The submission refers to Constitutional AI in the same way, as one of the published methods behind the agentic behaviors it describes, without detail on how (response to Question 2(a)). The receipts example also shows written instructions at the harness layer, such as a rule to flag charges over a hundred dollars, as a second kind of written control alongside training.

Limitations and open questions

• A policy document. The post describes Anthropic's practices and positions; it reports one statistic, attributed to separate research, and no new evaluation.
• Training effects are not isolated. The post attributes the doubling of check-ins on complex tasks to training, but the submission notes that product design also shapes the behavior.
• Threats are anticipated rather than observed. The submission states that its threat vectors come from research and red-teaming, not incidents.
• One company's evidence. The submission itself notes that "no single company's user base is representative" (response to Question 5(c)), which applies to the usage findings both documents cite.

Research questions

1. How much do the constitution and targeted scenarios each contribute to calibrated pausing? The post names both. Comparing models trained with only one of them on the same ambiguous agent tasks would separate their effects.
2. Does pausing track actual uncertainty? The submission says Claude is "at least partially calibrated." Comparing clarification requests with independent ratings of task ambiguity would measure calibration directly.
3. Does Plan Mode change what users catch? The post says plan-level review is where users most want to exercise judgment. Measuring how many seeded errors users find with plan review versus per-action approval would test whether it improves oversight.
4. How should failures without compromise be classified? The submission asks for a vocabulary. Collecting incident reports of agents acting within permissions but against intent, and testing whether reviewers classify them consistently, would show whether a shared taxonomy is workable.
5. How much does the environment layer reduce harm from successful injections? Running the same injection attacks against the same model in sandboxed and open environments, and measuring the damage achieved, would quantify the four-layer argument.
6. Do approval-prompt habits follow the pattern reported for Claude Code in other products? Repeating the auto-approval and interruption analysis in other agent products, as the submission suggests other companies should, would test how general the finding is.

How it connects

• Claude's Constitution is quoted in the post as a source of the training that teaches Claude to seek clarification.
• Measuring AI agent autonomy in practice is the research the post and the submission cite for check-in rates and approval habits.
• Building effective agents is the source of the definition of an agent used in both documents.
• Mitigating the risk of prompt injections in browser use is linked from the post's discussion of prompt injection.
• The submission cites Constitutional AI: Harmlessness from AI Feedback, Constitutional Classifiers, Constitutional Classifiers++, Sabotage evaluations for frontier models, SHADE-Arena and How AI is transforming work at Anthropic.

References

• Anthropic. (2026, April 9). Trustworthy agents in practice. https://www.anthropic.com/research/trustworthy-agents
• Anthropic. (2026, March 9). Response to Request for Information: Security Considerations for Artificial Intelligence Agents (Docket No. NIST-2025-0035), submitted to the NIST Center for AI Standards and Innovation. https://www-cdn.anthropic.com/43ec7e770925deabc3f0bc1dbf0133769fd03812.pdf
• Anthropic. (2026, January 21). Claude's Constitution. https://www.anthropic.com/constitution. Checked for the context of the quoted phrase (section "Having broadly good values and judgment").
• Anthropic. Our framework for developing safe and trustworthy agents (August 2025). https://www.anthropic.com/news/our-framework-for-developing-safe-and-trustworthy-agents. Summarized as the post describes it; not read for this note.