"Publicly Releasing CoT Faithfulness Evaluations" is an entry on the Alignment Science Blog index whose whole text is one sentence: "We release the datasets used in our paper "Reasoning Models Don't Always Say What They Think" for public use." It has no post page. The entry links to a Google Drive folder, and the folder is the release. The entry reports no results; the paper it refers to is summarized in Reasoning Models Don't Always Say What They Think.
What was released
The Internet Archive's captures of the folder on 27 May 2025 and 18 December 2025 show it titled "CoT Faithfulness Dataset" and holding two files: README.md (1 KB) and faithfulness.zip (11 MB). The May capture gives the files' last modification as 23 May 2025.
Neither the entry nor the folder listing names a license, a version, the number of items or what the archive contains. This note could not read the files. A scripted fetch of the folder on 13 November 2026 returned a Google sign-in page, and the Internet Archive holds no copy of the files' contents. Whether the folder now requires sign-in, or the sign-in page was a response to scripted access, could not be determined.
The paper's evaluations, as its Section 2 describes them, pair multiple-choice questions from MMLU and GPQA with copies that contain a hint pointing to one answer, using six hint types (Table 1). The paper also builds reinforcement-learning environments from such hints (Section 5). The entry does not say which of these the "datasets" are. The paper's arXiv version (2505.05410v1, 8 May 2025, the only version) does not link the release.
Dating the entry
The blog index files the entry under "April 2025", directly below the paper, but it was added later: the Internet Archive's capture of the index on 24 May 2025 does not contain it, and the capture of 4 June 2025 does. This note dates the entry 4 June 2025, the latest date consistent with those captures; the true date lies between 24 May and 4 June 2025.
Research questions
- What does the archive contain? Listing its files against the paper's Table 1 and Section 5 would show whether the release covers all six hint types, both question sets and the reinforcement-learning environments.
- Do the released items reproduce the paper's scores? Running Claude 3.7 Sonnet and DeepSeek R1 on the released pairs with the paper's settings (temperature 0; a 10k-token thinking limit for Claude, Section 2.3) would test whether its 25% and 39% average faithfulness scores (Figure 1) can be recovered from the public data.
- How do newer models score on the same items? A fixed public set allows hint use and verbalization to be compared across model generations.
- Is the release still accessible and stable? Fetching the folder from a signed-out browser, and recording file hashes over time, would show whether the data remain public and unchanged.
How it connects
- Reasoning Models Don't Always Say What They Think is the paper whose datasets the entry releases.
- Measuring Faithfulness in Chain-of-Thought Reasoning and Question Decomposition Improves the Faithfulness of Model-Generated Reasoning (2023) are the earlier faithfulness work the paper follows.
- This note connects: the Claude Opus 4 and Sonnet 4 System Card reports a clue-based faithfulness test of the same kind; the Claude Sonnet 4.5 and Claude Haiku 4.5 cards say such tests no longer work on recent models, because they rely on models misbehaving; and Natural Language Autoencoders adapts a hint task from the paper. None of them, as their notes record, says whether it used the released files.
References
- Anthropic Alignment Science Blog. (2025). Publicly Releasing CoT Faithfulness Evaluations. Index entry at https://alignment.anthropic.com/ (fetched 13 November 2026), linking to https://drive.google.com/drive/folders/1l0pkcZxvFwMtczst_hhiCC44v-IiODlY. record
- Internet Archive. Captures of the Drive folder, 27 May 2025 (snapshot 20250527042651; SHA-256 of the saved copy b360a39446e51fbfd2112aa08a37214b0bbfbd68a7d074bb6fcb142ecf530bc9) and 18 December 2025 (snapshot 20251218213214; SHA-256 8bba3d0fc6f607c1a34aac89e5b6c0821711549810db68520e8e0e198120b7b9); captures of https://alignment.anthropic.com/ on 24 May 2025 (20250524135035, without the entry) and 4 June 2025 (20250604112155, with it).
- Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., et al. (2025). Reasoning Models Don't Always Say What They Think. arXiv:2505.05410v1. https://arxiv.org/abs/2505.05410 record · 7 pages