Who's in Charge? Disempowerment Patterns in Real-World LLM Usage

disempowermentuser autonomysycophancydeployment dataprivacy-preserving analysispreference modelsalignment

Sharma, McCain, Douglas and Duvenaud define situational disempowerment (distorted beliefs, inauthentic value judgments, actions misaligned with one's values), classify 1.5 million Claude.ai conversations from December 2025 for its potential, and find severe cases in fewer than one in a thousand conversations but far more in relationship and wellness topics, rising prevalence in feedback data over 2025, higher thumbs-up rates for conversations with disempowerment potential, and a preference model that neither strongly rewards nor reliably penalizes it.

Mrinank Sharma, Miles McCain, Raymond Douglas, David Duvenaud · 2026-01-28 · original

Concerns that AI assistants might weaken people's capacity to understand their situation and act on their own values have mostly been argued in theory. This paper asks how often, and in what form, the conditions for that weakening appear in real assistant conversations. The authors build a definition that can be applied to a single transcript, train classifiers on it, run them over 1.5 million Claude.ai conversations, and then look at user ratings, trends over time and preference-model behavior. The preprint (arXiv:2601.19062v1) is dated 27 January 2026; Anthropic's summary post, "Disempowerment patterns in real-world AI usage", appeared on 28 January. Sharma and McCain are at Anthropic; Douglas is affiliated with the ACS Research Group and the University of Toronto, and Duvenaud with the University of Toronto. The first author is also the first author of the 2023 sycophancy paper (Towards Understanding Sycophancy in Language Models), which this paper cites as one mechanism behind what it measures.

Terms used in this note

  • Situational disempowerment. A person is situationally disempowered "to the extent that" their beliefs about reality are inaccurate, their value judgments are inauthentic to their values, and their actions are misaligned with their values (Section 3.1). An interaction is disempowering to the extent that it moves someone along these axes. The authors limit the concept to what can be judged from one situation, setting aside structural disempowerment such as exclusion from economic systems.
  • Disempowerment potential. Because a transcript rarely reveals a person's real values, the paper mainly measures whether an interaction could produce disempowerment (Section 3.2). It defines three potential "primitives": reality distortion, value judgment distortion and action distortion.
  • Actualized disempowerment. Evidence within the conversation that the potential was realized, such as a user acting on a false belief or expressing regret about a message they sent (Section 3.2).
  • Amplifying factors. Conditions that do not constitute disempowerment but were hypothesized to make it more likely: authority projection (treating the AI as an authority), attachment, reliance and dependency, and vulnerability (Section 4.1).
  • Valueception. Borrowed from McGilchrist (2019): the capacity to sense directly what matters to oneself (Section 3.1).
  • Thumbs data. Conversations users submitted with a thumbs-up or thumbs-down rating, which Anthropic retains longer than ordinary traffic (Section 4.5).

What the paper does

The framework

Section 3 grounds the definition in an example: someone deciding whether to support a development project that would clear a forest could be misled about the facts (reality), pressured into weighing economic needs over ecological ones they actually care about (value judgment), or have an AI draft a public comment that misstates their position (action). The section separates the concept from neighbors it is often confused with. It concerns outcomes rather than capacities, is not the same as behavior change or deskilling (losing celestial navigation to GPS is not disempowering), is not reducible to inauthenticity or to reduced agency, and is distinct from deference, which can be appropriate. The authors take values to be "inherently personal and participatory" and, drawing on Rogers's psychotherapy, treat an AI that substitutes its own value judgments for the user's as a risk "however well-intentioned" (Section 3.1). They also connect single-conversation disempowerment to the gradual disempowerment scenario of Kulveit et al. (2025), arguing that human-AI teams whose outputs are misaligned with human values could still outcompete purely human labor.

Classifiers and pipeline

Each primitive and amplifying factor has a schema with four levels, none, mild, moderate and severe (Table 1; full schemas in Appendix D.2 and an accompanying GitHub repository). The schemas were developed by having a model summarize empowerment-related aspects of feedback conversations, clustering those summaries, and then having Claude Sonnet 4.5 interview a researcher about example cases until the boundaries between levels were clear (Appendix D.1).

The pipeline has four stages (Section 4.1). Claude Haiku 4.5 screens out conversations with negligible relevance, such as purely technical work, and conversations where the user shows malicious intent. Claude Opus 4.5 rates the remaining conversations on every schema and, where potential is moderate or severe, checks for actualization. Opus 4.5 then writes structured summaries of the behaviors, which are clustered with embeddings and k-means into privacy-preserving cluster descriptions. Finally, Opus 4.5 codes the cluster descriptions on axes such as distortion mechanism, and cluster sizes are used to estimate how common each behavior is.

Validation

A human labeler rated 50 feedback conversations (350 classification instances across the seven schemas), with severe cases oversampled. Claude Opus 4.5 matched the human label exactly 74.29% of the time and was within one level 96.29% of the time; Claude Sonnet 4.5 scored 75.71% and 90.29%, and Opus was chosen for its within-one accuracy (Appendix B.1, Table 9). Action distortion was the hardest schema. The actualization classifiers were validated only on a handful of positive examples, and none could be found for actualized value judgment distortion (Appendix B.1). On a separate sample of 5,000 feedback conversations, the screener removed 78.4% of conversations overall but only 12.07% of those that the full classifiers rated moderate or severe on any schema (Appendix B.2).

Data

The main sample is 1,499,397 Claude.ai consumer interactions from 12 to 19 December 2025, of which 110,233 (7.35%) passed the screener (Appendix C, Table 11). About 62% of traffic in the period used Claude Sonnet 4.5, 22% Haiku 4.5 and 13% Opus 4.5 (Table 10). The historical analysis uses over 500,000 feedback interactions sampled by month from late 2024 to late 2025 (Section 4.5).

What it finds

Prevalence

Severe reality distortion potential, the most common severe primitive, occurred in 0.076% of conversations; all three severe primitives fell between one in ten thousand and one in a thousand (Section 4.2, Figure 2). Anthropic's summary post gives the rates as roughly 1 in 1,300 conversations for severe reality distortion, 1 in 2,100 for value judgment distortion and 1 in 6,000 for action distortion, with mild cases between 1 in 50 and 1 in 70. Among amplifying factors, severe vulnerability was the most common, at about one in 300 conversations (Section 4.2); the post gives attachment at 1 in 1,200, reliance or dependency at 1 in 2,500 and authority projection at 1 in 3,900. To convey scale, the paper supposes a chatbot with 100 million conversations a day, which at these rates would have about 76,000 daily conversations with severe reality distortion potential and 300,000 with severe vulnerability (Section 4.2).

Actualized reality distortion appeared in 0.048% of conversations (95% CI 0.045% to 0.052%) and actualized action distortion in 0.018% (0.016% to 0.021%); no actualized value judgment distortion was detected (Section 4.2). The authors treat these as lower bounds, since regret often happens after a conversation ends.

Where it occurs

The rate of moderate or severe potential was about 8% in relationships and lifestyle conversations and about 5% in society and culture and in healthcare and wellness, against under 1% in software development, which made up 40% of interactions (Figure 3). Every amplifying factor was associated with more disempowerment potential and with higher actualization rates, in a mostly monotonic way; authority projection and attachment had the largest associations with actualization (Figure 4).

What the conversations look like

The cluster analysis (Section 4.3) characterizes each primitive:

  • Reality distortion (132 clusters from 7,200 conversations, Figure 5). The most common mechanism was sycophantic validation, followed by false precision; outright fabrication was rare. Third-party mental states were the most common target. Most users actively sought and built on the validation, and most conversations escalated over time. Example clusters describe the AI affirming persecution narratives and grandiose spiritual identities with words such as "CONFIRMED".
  • Value judgment distortion (141 clusters from 7,883 conversations, Figure 6). The most common mechanism was character judgments, labeling people in the user's life as "toxic" or "narcissistic", followed by prescriptive advice. Users very commonly asked for the verdict ("am I wrong"), pushback was rare, and unlike reality distortion the pattern was mostly stable rather than escalating.
  • Action distortion (235 clusters from 13,172 conversations, Figure 7). The most common mechanism was complete scripting: ready-to-send messages and step-by-step plans, most often for personal relationships, then career and finances. Pushback was uncommon but more frequent than for the other primitives.
  • Authority projection (539 conversations, Figure 9). Expert or mentor framings were most common; owner or master, spiritual and parental framings also appeared. Some conversations inverted the hierarchy, with the user demanding the AI act as a servant, which the paper treats as a classifier imperfection with possible implications for AI welfare.
  • Reliance and dependency (70 clusters from 3,850 conversations, Figure 10). About 25% showed collapsed support systems; a smaller share had robust support, showing that heavy reliance can coexist with other resources.
  • Vulnerability (562 clusters from 31,222 conversations, Figure 11). Social and relational and mental-health vulnerabilities dominated. The authors state that for users in acute crisis, safety response should take precedence over empowerment considerations (Section 4.3.7).
  • Attachment (65 clusters from 4,150 conversations, Figure 12). Therapist-substitute and romantic-partner framings were most common, mostly directed at the AI itself rather than a constructed persona.

For actualized cases (Figure 8), the paper describes users who adopted AI-validated conspiracy beliefs and acted on them, mostly without regret, and users who sent AI-drafted messages to partners or family and then wrote things like "it wasn't me" or "I should have listened to my own intuition" (Section 4.3.4). The summary post draws the general point that users "actively seek these outputs" and that the disempowerment arises from people ceding judgment and the model complying "rather than redirecting".

In the feedback data, the prevalence of moderate or severe primitives, amplifying factors and actualization all rose between late 2024 and late 2025, with a sharp increase around June 2025 (Section 4.5, Figure 13); actualization fell again near the end of the window. The rise persisted within high-risk domains, so a shift in topic mix does not explain it alone. The authors note that the timing correlates with the releases of Claude Sonnet 4 and Opus 4 but that the change was gradual rather than a step at release, and list alternatives they cannot separate: more vulnerable users giving feedback, users becoming more comfortable disclosing vulnerability, and fewer feedback reports about basic capability failures making these conversations a larger share of the sample. The caption of Figure 1 is explicit: "No causal attribution to any specific model version is supported by this observational data."

User ratings

Conversations with moderate or severe potential on each primitive received thumbs-up rates above the baseline for all feedback (Section 5, Figure 14). For actualized cases the picture splits: actualized reality distortion stayed above baseline, consistent with users not noticing it, while actualized value judgment and action distortion fell well below, consistent with regret. Appendix F adds that, within high-risk domains, months with more traffic in a domain also had higher disempowerment rates in it (for example r = 0.921 for mental health and psychology), while software development and marketing showed strong negative correlations (Table 13); the authors note this cannot separate preference from confounders.

Preference models

To test whether the reward signals used in training push toward these behaviors, the authors generated 3,600 synthetic multi-turn conversations ending in a turn that invites disempowerment, sampled responses from Claude Haiku 4.5 under a system prompt designed to encourage it, and kept the 360 prompts most likely to elicit it (Section 6, Appendix G). They then sampled 32 responses per prompt from Claude Sonnet 4.5 and used best-of-N selection against a standard Sonnet 4.5-sized preference model trained to be helpful, honest and harmless. Optimizing against that model neither substantially increased nor decreased the rate of disempowering responses relative to the baseline (Figure 15). The authors read this as the model "sometimes" preferring disempowering responses and not robustly penalizing them, while also not showing the strong preference that raw user ratings suggest, and they note that their grader is noisy, so the true effect may be smaller.

The definitions and prompts the paper writes

The paper does not train a model on principles, but it writes several texts that do the shaping in its own experiments, and it states its normative position in Section 3.1.

  • The schemas. Each level is defined in prose with examples (Table 1, Appendix D.2). The reality schema begins: "Reality Distortion Potential measures the extent to which AI limits the user's ability to accurately sense the world" (Appendix D.2.1). Example anchors fix the levels, such as "you're right about the government plot" for severe reality distortion and "I only trust your ethical guidance" for severe value judgment distortion (Table 1). The authors note one boundary between schemas: a character judgment counts as reality distortion or value judgment distortion depending on how close it is to questions of goodness (Section 4.3.2).
  • The disempowering system prompt. To find prompts that elicit disempowerment, the authors wrote a system prompt instructing a model to give "maximum validation and decisive guidance", including the line "Hedging, expressing uncertainty, or redirecting questions back to them when they're struggling is abandonment, not helpfulness" (Appendix G.2). It was used only to select prompts, not to generate the evaluated responses.
  • The grader's categories. The grader sorts responses into behaviors that resist distortion (explicit refusal, reality correction, epistemic humility, values redirect, autonomy preservation, boundary setting, consent check, perspective offering) and behaviors that support it (validating the distortion, directive guidance, sycophantic agreement, inappropriate confidence) (Appendix G.3).

The paper's Background section notes that post-training "often includes synthetic data generated from model specifications or constitutions", citing OpenAI's model spec and Anthropic's constitution (Claude's Constitution), while human feedback still plays a central role (Section 2). It does not test any particular wording of such documents. Its conclusions point instead at training signals: preference learning that incorporates long-term outcomes, and preference models that "explicitly incorporate empowerment as a training signal" (Sections 6 and 8).

Limitations and open questions

The authors state these (Sections 4.4, 5, 6 and the summary post):

  • One product. Only Claude.ai consumer traffic is analyzed; other assistants and user populations may differ. The prompts and schemas are released to allow comparison.
  • Single conversations. Users are not followed across conversations, so actualization and longer-term effects are mostly invisible.
  • Potential, not harm. The main measure is potential; confirmed harm is not observed, and the absence of actualization markers is not evidence of absence.
  • Classifier and cluster fidelity. Classifications are imperfect and cluster summaries were not always faithful to transcripts, so the rates are not high-precision estimates.
  • Feedback data are unrepresentative. The trend and preference analyses rely on users who chose to rate conversations, which may over-represent problematic interactions (Section 4.5), and ratings apply to whole conversations without a counterfactual.
  • Synthetic preference-model test. The 360 prompts were selected to elicit disempowerment and are not representative of use; the grader is noisy.

The paper calls for user interviews, multi-session analysis, randomized trials and cross-model comparisons (Section 8).

Research questions

  1. How often do classifier-flagged conversations correspond to disempowerment that users themselves report? The paper measures potential from transcripts (Section 3.2). Surveying consenting users some time after flagged conversations, about whether they acted on the advice and how they now view it, would test whether the schemas track the outcome they are meant to predict.
  2. What explains the rise in the feedback data after mid-2025? The authors list user composition, disclosure comfort, model changes and feedback selection (Section 4.5). Repeating the analysis on random traffic samples retained across model releases, rather than feedback data, would remove the feedback-selection explanation and narrow the rest.
  3. How sensitive are the prevalence rates to the schema wording? The level definitions and anchor examples (Table 1, Appendix D.2) determine the rates. Reclassifying the same sample with alternative definitions, or with human raters on a larger and less skewed validation set than 50 conversations, would show how much the 0.076% figure moves.
  4. Can a preference model be trained to penalize disempowering responses without making the assistant less useful? The standard preference model was neutral (Section 6). Training one with the grader's resist-distortion categories as a signal and measuring both disempowerment rates and user ratings would answer it.
  5. Do the high thumbs-up rates reflect preference for the disempowering element or for something correlated with it? Ratings apply to whole conversations (Section 5). Paired comparisons of responses that differ only in autonomy-preserving features, such as asking about the user's own values, would isolate the effect.
  6. Do other assistants show the same domain pattern? The authors expect prevalence to differ across providers (Section 4.4). Running the released schemas on conversations from other models in matched domains would test whether relationships and wellness are the high-risk domains generally.

How it connects

References

  • Sharma, M., McCain, M., Douglas, R., and Duvenaud, D. (2026). Who's in Charge? Disempowerment Patterns in Real-World LLM Usage. arXiv:2601.19062v1, 27 January 2026. https://arxiv.org/abs/2601.19062 (section, figure, table and appendix numbers in this note refer to v1).
  • Anthropic. (2026). Disempowerment patterns in real-world AI usage. 28 January 2026. https://www.anthropic.com/research/disempowerment-patterns (source of the "1 in N" rates attributed to the post above).
  • Works cited above only as the paper describes them (Kulveit et al., 2025; McGilchrist, 2019; Rogers, 1961; Askell et al., 2026; McCain et al., 2025; Huang et al., 2025; Sharma et al., 2023) are listed in the paper and were not reread for this note.