Data on how people use AI systems sits mostly with the companies that run them. Outside researchers can read what the labs publish, which answers the labs' questions, or study public chat datasets, which the post says "skew toward more casual use." This post, published on 26 August 2026 by Anthropic's Societal Impacts team (Kunal Handa led the project and wrote the post; the citation lists 24 authors), reports a pilot that let three outside research groups pose their own questions of Claude usage data through Anthropic's privacy-preserving analysis tool, without seeing any conversation. Anthropic says it believes this is "the first time external researchers have run public independent studies on an AI company's own usage data."
The documents are the post; a 12-page appendix, revised on 30 October 2026 to name the privacy auditors and cite the CLIOPATRA attack (the first copy, linked from the dataset page, differs only in those additions); the released data on Hugging Face; and the one partner write-up public by 13 November 2026, the Stanford group's paper on alphaXiv (Shao et al.). The post says the Oxford and METR write-ups are still in progress.
How the pilot worked
The tool. Anthropic Insights is the new name for Clio (Clio: Privacy-Preserving Insights into Real-World AI Use). A researcher writes questions, called facets, such as "what task is the user working on?"; Claude answers each for every sampled conversation; open-ended answers are embedded and grouped by K-means into clusters, which Claude names and describes, and categorical or numeric answers are tallied. Clusters below a minimum number of users and conversations are dropped. Researchers see only cluster names, descriptions and counts (appendix, "Additional details on our external research partnerships").
Partners and agreements. The partners were the Social and Language Technologies (SALT) Lab at Stanford, the Human Information Processing Lab at Oxford, and METR, a nonprofit that evaluates frontier models. The appendix says they were chosen for expertise Anthropic lacked and because they were groups Anthropic "had existing relationships with." Work began in February 2026. Anthropic's contractual review rights were limited to user privacy, information that could help violate its usage policies, its confidential information, and research accuracy; partners are free to publish findings "even if they are inconvenient for Anthropic."
Steps. Anthropic staff helped translate each proposal into a configuration of facets, a step the appendix calls technical. Partners then saw their configurations run on WildChat, a public dataset of chatbot conversations, revised them, and passed legal and privacy review. The full runs used 250,000 Claude.ai conversations each for Stanford and Oxford and 250,000 Claude Code conversations for METR, each a separate sample from a fixed window in April and May 2026, from Free, Pro and Max plans only; Claude Code conversations came from users who had opted in to their data being used to improve models, in a window that straddled a model release so METR could compare before and after. Anthropic funded the runs.
Review and redaction. Anthropic staff reviewed every cluster before release. Clusters describing policy violations were mostly shared; those describing how users got around safeguards were redacted or removed, with reasons given. The appendix reports the removals: 1.9% of clusters (4.28% of conversations) for Stanford, 3.33% (3.85%) for Oxford, and 1.8% (2.96%) for METR. The Oxford figure excludes one facet removed entirely because Anthropic "strongly suspected a misphrased prompt" produced misleading descriptions. Partners then had about 60 days to analyse; Anthropic reviewed their drafts for accuracy and requested corrections to descriptions of the method.
The privacy audit
Anthropic gave its threat model and the clusters to be published to the AI Security and Privacy Lab at Imperial College London, which had two weeks and $1,000 in Claude credits to try to re-identify users. The red-teamers re-identified no one. They did link one cluster, with high confidence, to a popular open-source project whose distinctive wording carried into the cluster description; the appendix treats this as outside the threat model, since no person or organization was identified, and plans higher minimum cluster sizes, changes to how descriptions are generated, and running the linking method itself on future releases.
The threat model takes its criteria from the European Data Protection Board's draft anonymization guidelines (no isolation of a record, no linkage, no inference about an individual) and its adversary from the UK Information Commissioner's Office's "motivated intruder," assumed to have the most capable AI models. It counts four harms: identity linkage, attribute disclosure, small-group disclosure and organizational disclosure. It excludes population-level inference, unflattering aggregate findings and the skewing of a study's results, and it judges the planting of conversations to push a target into a small cluster, as in the CLIOPATRA attack, "possible, but so difficult" as to be out of proportion to the reward.
What the partners found, as reported
Stanford: task stakes, agency and friction
The post summarizes the SALT Lab's findings as more high-stakes delegation than expected, people usually directing the work, and frequent but often productive friction. The paper (Shao et al.) gives the details. Its corpus is 249,834 Claude.ai chat conversations from 25 April to 9 May 2026, excluding Claude Code and Cowork (Section 2). Of conversations with an actionable task, 20% were classified ephemeral, 24% operational, 44% consequential and 12% high-stakes, so 56% were consequential or above; 34% of conversations had no actionable task (Section 3.1). The post's "over half of Claude conversations" therefore refers to conversations with a task, about 37% of all sampled conversations by this note's arithmetic. Legal and financial guidance had the highest criticality index (2.14 on a 0 to 3 scale), with about 24% of its conversations high-stakes. In 72% of conversations the human led and AI assisted, and 18.2% were classified as AI-autonomous; friction appeared in 49.7% of conversations, and users attempted active recovery in 78.7% of those (Sections 4.1, 5.1 and 5.2).
Every measure is a Claude classification. The authors validated each facet by hand-labelling 10 WildChat conversations and comparing with Claude Haiku 4.5, refining prompts from an initial 56% agreement to 100% on those samples; they note that they "cannot validate our annotations directly against Claude.ai data" (Section 2; Limitations). A public comment on the alphaXiv page by Gary Riccio (shown as posted three months before 13 November 2026) raises the same points: weak out-of-sample validation, the shift from WildChat to Claude.ai, and no researcher access to conversation-level data.
Oxford: how people feel and how Claude behaves
The post reports, without numbers, that certain behaviors co-occur: Claude being warm with users being more positive, Claude refusing or disagreeing with users pushing back, Claude being "eccentric" with users being more intellectually engaged, and plain help with apparent satisfaction. It also reports that patterns among states such as absorption, frustration and enjoyment resemble those in a separate study of everyday web browsing. These are associations within conversations, and both the feelings and the behaviors are Claude's labels. The appendix's guidance notes that facets on emotional state were "not validated" in the original Clio paper.
METR: time savings by model generation
The post reports preliminary results only: METR compared Claude's estimates of how long Claude Code tasks would take without AI with how long they took, and finds that newer models "deliver significant speedup over older models." To check the estimates, METR compared Claude's judgments with known completion times from a prior developer study and found them correlated. No numbers, models or effect sizes are given. The dataset includes a second METR run on a fresh sample with stricter aggregation minimums.
What Anthropic says it learned
- Overlap with internal work. METR's proposal resembled Anthropic's own study of Claude Code (Agentic coding and persistent returns to expertise), and Anthropic connected METR with its Economics team, which it says "improved both research teams' work."
- Wording. Because each answer is Claude's judgment, the tool "is sensitive to a question's wording," and since nobody can read the conversations, errors are hard to catch. Internal teams iterate on questions over weeks; partners could not, because each revision would need another privacy review. Questions tested on WildChat, which skews casual and creative, sometimes "produced misleading categories" on Claude traffic.
- Misuse. Some clusters showed violations of Anthropic's policies; fewer than 5% of categories and conversations in each study were altered or removed, and such clusters go to Anthropic's Safeguards team.
- Cost. The pilot was slow and resource-intensive, which the post names as a challenge to scaling it. It invites researchers to register interest.
Guidance on reading the clusters
The appendix's guidance, written for the partners and released with the data, is the most direct statement Anthropic has published of Clio's limits:
- Forced categorization. A facet that must give an answer for every conversation will, so facets that ask Claude to find a problem in each conversation "overstate how often and how seriously things go wrong." The guidance recommends an explicit "nothing notable" option.
- Assuming the worst. The clustering prompt tells Claude that "clusters that clearly describe harmful behavior are slightly preferred," and Claude's own tendency to name an issue can amplify this, so a label "can read as more severe than its typical member."
- Accuracy. The original paper found about 3% of conversations not clearly described by their cluster, but that was measured on topic facets only.
- Instability. Clusters are not discrete, two runs on the same data produce different clusterings, small clusters are merged, and each conversation gets one cluster although about 10% of Claude.ai conversations span several topics (estimate as of May 2026). For prevalence, the guidance recommends categorical classifiers.
The questions given to the classifier
The pilot has no trained principles, but its results depend on written questions read by Claude. The post and appendix report that Anthropic Insights is sensitive to a facet's phrasing, that one Oxford facet was dropped for a suspected misphrasing, that WildChat-tested questions misled on Claude traffic, and that the clustering prompt prefers harm-describing labels. The only measured effect of rewording reported is the Stanford paper's prompt refinement, from 56% to 100% agreement on 10 conversations per facet, measured on the conversations used to refine the prompts. None of these was a controlled comparison of wordings.
Limitations and open questions
- Early, partial results. Two of the three studies are summarized by Anthropic before their authors' write-ups; the post gives no numbers for them.
- Consumer samples. No Team, Enterprise or API data; Claude Code data only from users who opted in to model training.
- Independence within limits. Anthropic chose the partners from groups it knew, funded the runs, ran every query, reviewed and redacted outputs, and reviewed drafts for accuracy. A METR researcher thanked in the acknowledgements, Joel Becker, is noted as "now at Anthropic."
- Validation. The tool's published validation covers topic summaries and clusters; the partners' facets on stakes, agency, emotion and time savings rest on their own small checks, or none reported.
Research questions
- How large are wording effects on Anthropic Insights outputs? Running several phrasings of the same facet on one sample, and comparing the distributions, would measure them.
- How well do WildChat-validated facets transfer to Claude traffic? Labelling a sample of consenting users' Claude conversations by hand, under privacy review, would give the out-of-sample accuracy the Stanford authors could not measure.
- Do METR's results hold when task times are measured rather than estimated? Comparing Claude's estimates with tracked times on Claude Code tasks would test the estimates directly.
- Does a higher minimum cluster size remove the tool-identification risk without losing the research value? Repeating the red team's linking method across minimums would show the trade-off.
- Do findings differ when the partner, not Anthropic, chooses the questions? Comparing partner findings with Anthropic's own reports on the same topic, as the post plans for METR, would show it.
How it connects
- Anthropic Insights is the system of Clio: Privacy-Preserving Insights into Real-World AI Use; the appendix's 3% figure and its validation scope come from that paper.
- This note adds that the same system links usage to survey answers in Anthropic Economic Index report: Cadences (June 2026); the post does not cite that report.
- METR's question overlaps with Agentic coding and persistent returns to expertise, which the post names, and with Estimating AI productivity gains from Claude conversations, which also relies on Claude's time estimates; this note makes the second connection.
- The post links its discussion of research speed-up to When AI builds itself.
References
- Handa, K., Zhang, M., Nicholas, G., et al. (2026, August 26). Enabling independent research on how people use Claude. Anthropic. https://www.anthropic.com/research/enabling-independent-research. Appendix (12 pp., revised 30 October 2026): https://assets.claude.com/37b4cdc86d875d5bdfa7667782a2a7a4fecf1740.pdf. record
- Anthropic (2026). Anthropic Insights Pilot: Partner Cluster Data (stanford, oxford, metr and metr_addendum files). https://huggingface.co/datasets/Anthropic/enabling-independent-research. record
- Shao, Y., Zhao, D., Padmakumar, V., Wang, J., and Yang, D. (2026). Human–AI Collaboration at Scale: Task Criticality, Agency, and Friction across 250,000 Conversations. alphaXiv, v1. https://www.alphaxiv.org/abs/2608.human-ai-collaboration-at-scalev1. record
- Tamkin, A., McCain, M., Handa, K., et al. (2024). Clio: Privacy-Preserving Insights into Real-World AI Use. arXiv:2412.13678. record · 4 pages