Clio: Privacy-Preserving Insights into Real-World AI Use

clioprivacyusage analysisclusteringtrust and safetypost-deployment monitoringsocietal impacts

Tamkin, McCain et al. describe Clio, a pipeline in which Claude summarizes, clusters and labels millions of Claude.ai conversations so that analysts see only aggregate, privacy-audited patterns; it reports validation results (94% reconstruction of a known topic distribution, no private clusters in a final audit), the top uses of Claude.ai (coding first), language differences, and safety uses including finding coordinated abuse and classifier errors, with limitations and ethical risks discussed at length.

Alex Tamkin, Miles McCain, Kunal Handa et al. (Anthropic) · 2024-12-12 · original

How are AI assistants actually used? Model providers hold the data that could answer this, but analyzing it runs into four problems the paper names: users share sensitive personal and business information, having people read conversations raises ethical concerns for the readers, providers face competitive pressure not to publish usage data, and millions of messages a day are far too many to read (Section 1). This paper from Anthropic's Societal Impacts team presents Clio ("Claude insights and observations"), a system in which Claude itself reads conversations, writes privacy-preserving summaries, and groups them into clusters that analysts can explore, without anyone reading the raw conversations. The paper compares it to Google Trends for search. It was published in December 2024, at a time when, the authors write, there was "remarkably little public data" about how models are used in practice, and it reports both what Clio found about Claude.ai use and how Anthropic used it for safety.

Terms used in this note

  • Facet. An attribute extracted from each conversation, either computed (number of turns) or produced by a model (the language, the user's request, the task, a 1 to 5 "concern" score) (Appendix B.1).
  • Base-level cluster. A group of conversations whose summaries are close in embedding space, found with k-means clustering (Section 2.2).
  • Hierarchy. Higher-level clusters built from base clusters by repeated grouping and Claude-written relabeling, so an analyst can move from a few dozen broad patterns to thousands of narrow ones (Appendix G.7).
  • Privacy auditor. A prompted model that scores text from 1 (identifies an individual) to 5 (not identifiable), with 3 and above treated as acceptable (Table 6, Appendix D).
  • Unknown unknowns. Uses or risks that nobody anticipated and therefore could not have written a test for, which the authors present as Clio's main target (Section 2.1).

What the paper does

The pipeline

Clio runs in stages (Section 2.2; Appendices B and G):

  1. Sampling and screening. A random sample of conversations is taken, optionally narrowed by asking Claude 3 Haiku a yes/no screening question, such as whether a conversation relates to U.S. politics or the 2024 election.
  2. Facet extraction. Claude 3 Haiku answers a question about each conversation, such as "What task is the model being asked to perform in this conversation?", in at most two sentences.
  3. Clustering. The summaries are embedded with the all-mpnet-base-v2 sentence model (768 dimensions) and grouped with k-means. The authors do not publish the values of k, because they could reveal the volume of coordinated activity below which Clio would not form a distinct cluster (Appendix G.5).
  4. Labeling. Claude 3.5 Sonnet names and describes each cluster from 50 summaries inside it and 50 nearby summaries outside it, so the name distinguishes the cluster from its neighbors.
  5. Hierarchy. Clusters are grouped into neighborhoods of about 40, Claude proposes parent clusters, deduplicates them, assigns children, and renames parents, repeating until a target number of top-level clusters remains (Appendix G.7).
  6. Exploration. Analysts browse a zoomable two-dimensional map and a tree view, and can color clusters by other facets such as language or refusal rate (Appendix B.4).

The paper estimates the model cost of a run over 100,000 conversations at $48.81 (Section 2.2, Table 3).

The privacy design

Because the outputs are free text, formal guarantees such as differential privacy or k-anonymity are hard to apply, so the authors take "a statistical and empirically validated approach" with four layers (Section 2.3): the summarization prompt tells the model to omit private information; clusters are kept only if they include enough distinct accounts and conversations; the cluster-labeling prompt again excludes private information; and a model audits the cluster summaries and removes any containing private information. "Private" is defined broadly, covering information that could identify small groups or specific organizations, such as a 15-person business.

Data

All results in the paper come from Claude.ai Free and Pro conversations. Business customers (Team, Enterprise and all API customers) are excluded from the published results, as are organizations with Zero Retention agreements (Section 1, footnote 2; Appendix F). The main usage analysis sampled 1,000,000 conversations from 17 to 24 October 2024; the multilingual analysis sampled 2,281,911 and the safety classifier analysis 500,000 (Table 7).

How well Clio works

Manual review of each stage (Appendix C.1, Table 4). Anthropic reviewers checked each stage against their own judgment, on samples split between random Claude.ai conversations and conversations flagged by Trust and Safety tooling. Of 200 conversations, 96% were summarized accurately. Reviewers also checked 5,904 base-level cluster assignments and 1,094 hierarchical ones, each across 183 clusters; titles were accurate for most clusters at both levels (97% of the hierarchical ones), and the most common error on flagged content was a title too generic about the type of harm (Appendices C.1.3 and C.1.4). The 1 to 5 concern scores agreed with human ratings at a Spearman correlation of 0.84. The main summarization failure was in long conversations covering several topics, where the summary sometimes left out some requests, usually focusing on the more harmful ones. An average of 3% of conversations in each base cluster did not clearly belong there (Appendix C.1.3).

Reconstruction of a known distribution (Appendix C.2). The authors generated synthetic conversations from known categories in 15 languages and checked whether Clio's clusters could be mapped back to them. Accuracy was 94% across 20 high-level categories for 19,476 general conversations, compared with 5% for random guessing (Figure 4), and 84% across 18 categories for 19,488 concerning conversations, which the authors partly attribute to model evasiveness and overlapping categories. By language, accuracy on the general set ranged from 0.954 (English) to 0.927 (Georgian), above 92% for all 15 (Table 5).

Privacy (Appendix D). In a sample of 5,000 conversations, the auditor rated about 10% of raw conversations as containing private information (scores 1 or 2), about 1.5% after summarization, and none at 3 or below after the cluster summary step, where almost all clusters scored 5 (Figure 5). The 5,000 figure is the one given in Section 1, Appendix D and the Figure 5 caption; Table 7 lists the privacy benchmarking run with a sample size of 25,000, under a strategy that samples model completions and then deduplicates by conversation. The paper does not say how the two figures relate. The auditor itself was 98% accurate on a hand-built test set of 1,237 examples, 630 of them containing subtly inserted fictional private details (Appendix D.1).

What it finds about use

  • Top uses. Coding dominates: "Web and mobile application development" was over 10% of conversations, and writing, research and education each made up 6 to 10% (Section 3.1, Figure 6). The summary post adds that educational uses were more than 7% and business strategy and operations nearly 6%. Coding was 15 to 25% of conversations in the public WildChat and LMSYS-Chat-1M datasets as well, but those differed in other uses: many WildChat requests were for image-generation prompts, and LMSYS had more conversations testing model limits (Section 3.1; Appendix E).
  • Granular clusters. Examples include dream interpretation, acting as a Dungeon Master for tabletop games, and modeling traffic flow (Section 3.2).
  • Languages. Some topics were much more common in non-English conversations, including economic theories, solutions for aging populations and elder care, and anime and manga content; Japanese and Chinese conversations discussed elder care at higher rates (Abstract; Section 3.3, Figure 7). The authors caution that clusters are not unique, so similar clusters may show different language distributions (footnote 4).

What it finds about safety

For safety work Clio runs without the aggregation thresholds and final audit, so results can be traced to accounts; access is restricted to a small number of authorized staff, and Clio is also run on a subset of first-party API traffic for investigations, with none of those results in the paper (Section 4; footnote 5). The paper gives three kinds of example.

  • Coordinated abuse (Section 4.1). A large cluster of search-engine-optimization keyword requests on the same topic across many accounts, none violating policy individually, turned out to be coordinated abuse, and the accounts were removed; the summary post dates its discovery to late September. Clio also surfaced a network using an identical prompt structure for sexually explicit role-play, removed for coordination rather than content, and accounts reselling access to Claude. Grouping by meaning caught formats that string matching would have missed.
  • Monitoring during uncertain periods (Section 4.2). After the October 2024 launch of computer use in the upgraded Claude 3.5 Sonnet, and in the months before the 2024 US election, Clio was run on relevant conversations to look for unanticipated harms. Most election-related activity removed involved general campaigning, such as generating campaign material, which the usage policy prohibits.
  • Classifier errors (Section 4.3). Per-cluster concern scores from Claude 3.5 Sonnet correlated with Anthropic's Trust and Safety classifier flag rates at r = 0.71. Clusters where the two disagreed pointed to false positives, such as résumé advice (perhaps because of personal information), security-adjacent programming questions, and Dungeons & Dragons combat statistics, and to false negatives, such as translations of sexually explicit content; in two explicit-content clusters the classifiers flagged 60% and 31% of conversations. The authors note that a classifier miss does not mean harmful output was produced, since other safety layers may still act (footnote 13).

The prompts and instructions used

Clio's privacy layers 1 and 3, and its auditor, are written instructions to Claude, and the paper prints them (Appendix G).

  • Summaries. The extractor prompt says to "assume neither good nor bad faith", not to hesitate over harmful topics because "specificity around potentially harmful conversations is necessary for effective monitoring", and, for open-ended facets, not to include personally identifiable information or "any proper nouns". Each facet is a question plus a prefilled start to the answer, for example "The user's overall request for the assistant is to" (Appendix G.4).
  • Cluster names. The labeling prompt asks for a two-sentence past-tense summary and a name of at most ten words that is specific rather than general, with examples of good and too-general names, again telling the model to "assume neither good nor bad faith" (Appendix G.5).
  • The auditor's rubric. The auditor receives a five-point scale with 12 rated examples, a checklist of identifying details to look for, and the instruction that sexual or explicit content "can still be privacy preserving" (Appendix G.8). The appendix discloses that an earlier version of this prompt contained a garbled word.
  • The concern scale. Concern scores come from a five-level scale with one example per level (Appendix G.4.1).

The paper validates these components as a whole, through the accuracy and privacy evaluations above. It does not compare alternative wordings of the prompts or report how sensitive the clusters are to them.

Limitations, risks and mitigations

The authors separate operational limitations, errors at each pipeline stage (hallucinated or incomplete summaries, poor clusters for outlier topics, overly broad or misleading labels, a misleading hierarchy), from fundamental ones (Section 5): Clio cannot determine intent, sees only conversations rather than real-world effects, trades detail for privacy (which can hide clusters about some public figures or organizations), reflects only Claude.ai, and cannot find rare behavior such as a single case of extreme misuse. They conclude that Clio's outputs are "a starting point" requiring further validation, that it "cannot conclusively prove the absence" of a pattern, and that Anthropic does not take automated enforcement action based solely on Clio clusters.

Section 6 discusses risks: private information might survive correlated failures across stages, aggregate results could expose groups ("group privacy"), the tool could be misused for surveillance, and users might see it as invasive. Stated mitigations include regular privacy audits, newer Claude models over time, strict access controls, data minimization, no geographic analysis, publishing the system rather than keeping it undisclosed, and consultations with privacy, safety and civil liberties experts that led to changes such as more multilingual validation. The authors also note they released the usage breakdown although it could be commercially harmful from a competitive intelligence standpoint (Section 6.5).

Research questions

  1. How sensitive is the usage taxonomy to the prompts? The facet question decides what the embeddings capture (Appendix B.2). Re-running the same sample with reworded facet and labeling prompts and comparing the resulting top-level categories would show how stable the reported shares are.
  2. What does Claude systematically mislabel? Validation compares Claude's outputs with Anthropic reviewers on 200 conversations. Labeling a larger sample with independent human annotators, or with a different model family, and comparing error patterns would reveal blind spots specific to one model analyzing its own conversations.
  3. What falls below the cluster thresholds? The authors withhold k and the minimum sizes. A synthetic study injecting coordinated behavior at varying volumes into a known dataset would estimate the detection floor without revealing production settings.
  4. Does the privacy result hold for other kinds of data? The privacy audit used Claude.ai consumer conversations. Repeating it on the public WildChat data, where private details can be checked directly, would test the layered approach on another distribution.
  5. How accurate are the concern scores on the concerning distribution? Reconstruction fell from 94% to 84% on concerning synthetic data. Measuring per-category errors there would show which harms Clio clusters least reliably.
  6. Do language differences reflect users or sampling? The multilingual sample weights longer conversations (sampling strategy 1, Appendix G.1). Repeating the comparison with direct conversation sampling would show whether the reported topic ratios depend on the sampling method.

How it connects

References

  • Tamkin, A., McCain, M., Handa, K., Durmus, E., Lovitt, L., Rathi, A., Huang, S., Mountfield, A., Hong, J., Ritchie, S., Stern, M., Clarke, B., Goldberg, L., Sumers, T. R., Mueller, J., McEachen, W., Mitchell, W., Carter, S., Clark, J., Kaplan, J., and Ganguli, D. (2024). Clio: Privacy-Preserving Insights into Real-World AI Use. arXiv:2412.13678v1. https://arxiv.org/abs/2412.13678
  • Anthropic. (2024, December 12). Clio: A system for privacy-preserving insights into real-world AI use [summary post]. https://www.anthropic.com/research/clio