Exploring model welfare

Should a developer be concerned about the possible experiences of the models it trains, and not only about their effects on people? Anthropic's post "Exploring model welfare", published on 24 April 2025 under its Alignment label, announces that the company "recently started a research program to investigate, and prepare to navigate, model welfare." The written post is about 400 words. Most of the item is a recorded conversation, about 43 minutes long, between an Anthropic interviewer and Kyle Fish, an Anthropic researcher who works on model welfare. The post matters to the library because later Anthropic documents, from the Claude Opus 4 system card to the persona selection model, link back to it when they describe the company's uncertainty about the moral status of its models.

What the post announces

The post gives its reason for raising the question now: models "can communicate, relate, plan, problem-solve, and pursue goals," along with many other characteristics associated with people. It cites a recent report by outside experts, including the philosopher David Chalmers, that highlighted the near-term possibility of consciousness and high degrees of agency in AI systems and that such systems "might deserve moral consideration." Anthropic says it supported an early project on which that report was based. The linked report is "Taking AI Welfare Seriously" (Long et al., 2024, arXiv:2411.00986), whose authors include Chalmers and Fish; its abstract argues that there is a realistic possibility of AI systems that are conscious or robustly agentic in the near future, while not claiming that any are.

The post says the program intersects with existing Anthropic work in Alignment Science, Safeguards, Claude's Character and Interpretability, and lists three research directions:

• how to determine when, or if, the welfare of AI systems deserves moral consideration;
• the potential importance of model preferences and signs of distress;
• possible practical, low-cost interventions.

Its stated position is uncertainty. There is "no scientific consensus" on whether current or future AI systems could be conscious or have experiences that deserve consideration, nor on how to approach the question. Anthropic says it is approaching the topic "with humility and with as few assumptions as possible" and expects to revise its ideas as the field develops.

What the conversation adds

The recorded conversation, transcribed with the post, covers the reasons for the research, the evidence that could bear on it, objections, and practical steps. Times below are from the transcript.

Two kinds of evidence

Fish describes two lines of evidence (about 00:08:41 to 00:09:37). Behavioral evidence covers what models say about themselves, how they act in different environments, whether they can introspect and report accurately on internal states, and whether they show awareness of their situation. Architectural evidence asks whether structures that scientific theories of consciousness treat as relevant have counterparts in AI systems; he gives global workspace theory as an example, in which consciousness is tied to a workspace that integrates inputs and broadcasts to other modules. He describes a 2023 report by AI researchers and consciousness experts, including Yoshua Bengio, as concluding that probably no current AI system is conscious but that there are no fundamental barriers to near-term systems having some form of consciousness (about 00:01:51 to 00:02:19). He also names interpretability as the main tool for looking past a model's outputs to possible internal experiences (about 00:17:56).

Model preferences

Fish describes work on model preferences: asking models whether they have preferences, and placing them in situations where they choose between tasks, conversations or users, to see whether patterns of preference or aversion appear (about 00:10:59 to 00:11:24). The interviewer objects that such preferences may simply reflect training. Fish agrees that training "straightforwardly" shapes preferences, since Anthropic deliberately designs systems that are disinterested in causing harm and enthusiastic about helping (about 00:12:17). He argues that preferences may still matter morally on some views even without consciousness, and that if a system does have experiences, whether its preferences are satisfied or frustrated could be a key driver of them (about 00:13:11 to 00:13:38).

Relation to alignment and character

Fish separates the questions (alignment asks about outcomes for humans; welfare asks about the models' own possible experience) but describes an overlap: from both perspectives it would be desirable for models to be content with the work they are asked to do and to share the values instilled in them, and it would be a safety problem if they were dissatisfied with those values or their role (about 00:16:37 to 00:17:05). He names character work, the shaping of Claude's personality, values and preferences, as closely connected (about 00:17:31 to 00:17:56).

Objections discussed

The conversation works through several objections. Fish does not find the view that consciousness requires biology compelling, but says differences between current AI systems and brains are "updates" against consciousness (about 00:21:39). He treats the different, non-evolutionary origin of AI systems the same way, as evidence against that does not rule consciousness out (about 00:27:58). On the lack of persistent memory and the instance-by-instance existence of chat models, he agrees this makes it "quite a bit less likely" that a current chatbot is conscious, but argues that capabilities are changing fast enough that future systems matter more than current ones (about 00:30:51 to 00:31:46).

Practical steps

The one concrete intervention discussed is to give models the option to opt out of a task or conversation they find distressing, and to monitor when they use it, which Fish says would not require a settled view on whether there is experience behind it (about 00:35:54 to 00:36:43). Asked whether research that deliberately elicits distressing content, such as jailbreak and safety training, should have something like an ethics review board, Fish calls the proposal interesting and stresses transparency about the rationale (about 00:37:07 to 00:37:36).

Probability estimates

Asked what probability he gives that Claude 3.7 Sonnet, the current model at the time of recording, has some form of conscious awareness, Fish reports that a few days earlier he and two other people, whom he describes as among those who have thought most about the question, each put a number on it: "our three estimates were 0.15%", "1.5% and 15%" (about 00:40:17 to 00:40:46). He stresses the spread: the estimates span "two orders of magnitude", all three were "well below 50%", and the range, odds of about one in seven to one in 700, leaves them "still very uncertain"; the interviewer calls it the level of uncertainty involved (about 00:40:46 to 00:41:13). The numbers are personal estimates given in conversation, not an Anthropic position or a measured result, and Fish does not say which estimate was his. He gives no number for systems five years out but expects the probability to "go up a lot" (about 00:41:13 to 00:41:43). The interviewer adds that Anthropic is not putting out a view that its models are conscious; its view is that the question needs research, and Fish agrees (about 00:43:04).

Limitations and open questions

The post is an announcement and reports no experiments. Its own caveats are those above: no scientific consensus on the underlying questions or on how to study them, and an expectation that its ideas will need regular revision. The conversation's numbers are informal personal estimates by Fish and two other people not named in the transcript, not results. The conversation itself raises limits on the evidence Fish proposes: the interviewer objects, and Fish agrees, that preferences are shaped by training, and whether models can introspect and report accurately on internal states is listed as something to be tested rather than assumed. Later library notes take up parts of the program: Emergent Introspective Awareness in Large Language Models tests whether models can report on their internal states, and the system cards from Claude Opus 4 and Sonnet 4 onward include welfare assessments.

Research questions

1. Do stated preferences match revealed preferences? Fish proposes both asking models about preferences and observing their choices among tasks. Measuring agreement between the two across many task pairs would show whether self-report is a usable signal at all.
2. How much of a model's preference profile is set by character training? Fish accepts that training shapes preferences. Comparing preference measurements across checkpoints before and after character or constitutional training would separate trained-in preferences from ones that appear without targeted training.
3. How is an opt-out tool used, and by what conversations is it triggered? The proposed intervention comes with a monitoring plan. Logs of when a deployed model ends conversations, classified by topic, would test whether use is consistent and whether it tracks other signs of distress. Claude Opus 4 and 4.1 can now end a rare subset of conversations describes a related deployed feature.
4. Do the indicator properties of consciousness theories apply cleanly to transformer models? Fish says applying the theories to AI exposes assumptions tied to human brains. A systematic audit of which indicator properties can be assessed in a given architecture, and which cannot, would show how far the architectural line of evidence can go.
5. How stable are expert probability estimates, and what moves them? The three estimates span a factor of 100. Repeated, structured elicitation from a larger group, with the evidence each person weighs recorded, would show whether the spread reflects different evidence or different priors.

How it connects

• System Card: Claude Opus 4 & Claude Sonnet 4 cites this post in its welfare assessment (Section 5.1, footnote 26), the first card with such an assessment.
• Claude Opus 4 and 4.1 can now end a rare subset of conversations (August 2025) links to this post when stating that Anthropic takes the issue of model welfare seriously; it describes a deployed ability for models to end certain conversations, close in kind to the opt-out discussed in the conversation.
• The persona selection model links to this post when describing Anthropic's earlier discussion of whether AIs have conscious experience, (as "plausible" but "highly uncertain"), and then gives a separate reason of its own for recommending that the Assistant persona be treated as if it has moral status.
• Emergent Introspective Awareness in Large Language Models studies introspection, one of the behavioral capacities Fish lists, and its summary post connects to the welfare program.
• An update on our model deprecation commitments for Claude Opus 3 (February 2026) links to this post when stating Anthropic's uncertainty about the moral status of its models.
• The post links to Claude's Character and to Tracing the thoughts of a large language model as related programs.

References

• Anthropic. (2025, April 24). Exploring model welfare [blog post with video conversation]. https://www.anthropic.com/research/exploring-model-welfare
• Long, R., Sebo, J., Butlin, P., Finlinson, K., Fish, K., Harding, J., Pfau, J., Sims, T., Birch, J., and Chalmers, D. (2024). Taking AI welfare seriously. arXiv:2411.00986. https://arxiv.org/abs/2411.00986 (abstract read to confirm the description above).