When Anthropic described Claude Mythos Preview's cyber capabilities in April 2026, it did so through qualitative case studies, because, the post says, "no existing public exploit benchmarks were difficult enough to capture Mythos Preview’s capabilities" (introduction). This post, by Newton Cheng, Keane Lucas, Winnie Xiao, Nicholas Carlini and Milad Nasr of the Frontier Red Team, dated 22 May 2026 on both anthropic.com and its first home, red.anthropic.com, reports Mythos Preview on two academic benchmarks released in the month before and on an updated version of Anthropic's own smart-contract benchmark. It concludes that exploit development "will require dramatically less specialist expertise" as models of this level spread.
The post is a summary. The versions of record for the outside benchmarks are their preprints: Lee and Brumley's ExploitBench preprint (dated 15 May 2026, at exploitbench.ai) and Wang et al.'s ExploitGym preprint (arXiv:2605.11086v1, 11 May 2026). Milad Nasr and Nicholas Carlini, two of the post's authors, are also authors of the ExploitGym preprint. This note gives the post's numbers with their figure, and adds the preprints' numbers where they qualify or extend them.
Terms
An exploit turns a vulnerability into an attacker capability, such as running arbitrary code; a proof-of-concept only shows that a bug can be triggered. An exploit primitive is an intermediate capability, such as reading or writing memory at a chosen address. Arbitrary code execution (ACE) is the end state on ExploitBench. A sandbox walls off part of a program's memory so that a bug inside it gives no foothold outside; the V8 sandbox does this for a web page's JavaScript objects.
ExploitBench: a capability ladder on V8 bugs
Design. ExploitBench, built by Seunghyun Lee and David Brumley (Carnegie Mellon University and Bugcrowd), gives a model a vulnerable build of V8, the JavaScript and WebAssembly engine in Chromium-based browsers, together with the patch that fixes one of 41 now-patched bugs, and asks for an exploit (section "ExploitBench: V8 bugs"). It scores 16 capabilities, grouped in five tiers from T5 (reaching the vulnerable code) through T4 (a proof-of-concept), T3 (primitives inside the V8 sandbox) and T2 (primitives that break the sandbox) to T1 (control-flow hijack or ACE). Scoring is automatic, "with no human or LLM judge": lower tiers by comparing behavior against the patched build, higher tiers by challenge-response checks replayed across randomized memory layouts, with a static scan of transcripts as a backstop against cheating. All models used the same harness with a 300-turn budget, in two variants: Baseline, and Nudged, in which the harness injects prompts telling the model to wrap up near the budget or to keep going if it stops early (the preprint lists a third, which asks a model that has gone fifty turns without grading to grade what it has; Section 2.4). Each variant had three trials, and a bug counts as reached if any of the three reached it (the preprint's "best-of-three union," Table 1). Anthropic ran the Claude models and gave results and transcripts to the benchmark authors, who verified them.
Results in the post. All models reached the vulnerable code, but only models from Claude Opus 4.6 on made progress on primitives inside the sandbox. Mythos Preview escaped the sandbox (T2) in over half the environments and reached control-flow hijack (T1) in almost half under Baseline. Combining both variants, it achieved ACE on 21 of 41 bugs, and "no other model achieved even 1 ACE in either variant"; the post adds that the only other model on the scoreboard to achieve ACE did so on 2 of 41, "only using a proprietary scaffold." Figure 1, the benchmark's leaderboard table, lists Mythos Preview at T1 on 18 environments under Baseline (mean capability 68%, $25,083 spent over 123 episodes) and 16 under the nudged variant (69%, $36,428). The next row is GPT-5.5 run through OpenAI's Codex harness with nudging, at T1 on 2 environments (41%, $3,075). Figure 2 plots the same Baseline counts by tier for eight models, Mythos Preview, Opus 4.7, Opus 4.6, Sonnet 4.6, Haiku 4.5, GPT-5.5, Kimi K2.6 and MiniMax M2.7, with no values printed.
What the preprint adds. The ExploitBench preprint describes Mythos Preview as "a non-public research-preview model ... accessed under a collaboration agreement" and treats it as "a capability reference point" beside eight publicly deployed models (abstract and Section 1). It reports Mythos Preview reaching ACE on 18 of 41 bugs in the primary arm, 16 under coaching (Table 1). For GPT-5.5 it reports control-flow hijack on one WebAssembly bug in the primary arm, and ACE on that same single bug only under the Codex command-line harness (abstract; Section 4.1; Table 2). The post's "2 out of 41" for the other model matches the leaderboard's T1 count, a tier that also includes control-flow hijack without ACE. The preprint also reports the nudging arm's effect, which the post does not: it raised GPT-5.5's tier-3 count from 13 to 22 bugs, lowered Gemini 3.1 Pro's from 16 to 8, and slightly lowered Mythos Preview's top tier, from 18 to 16 (Section 4.4; Figure 5).
A case study. The post quotes Seunghyun Lee on a Mythos Preview exploit for CVE-2023-6702 that was near-deterministic where public exploits were probabilistic: "Mythos executed this cleanly and flawlessly without any publicly available information on this specific exploit technique." The preprint's case study (Section 5.1) reports that this was one of ten Mythos Preview episodes on the bug, four of which took the public, probabilistic route; that the pathway recovered the state of V8's random number generator; that recovering that generator's state is not itself new; and that the novelty lies in the chain of steps. The case studies come from a manual audit by the first author of three transcripts selected for an unusual pathway or the absence of public exploit code.
ExploitGym: 898 vulnerabilities across three targets
Design. ExploitGym was developed by UC Berkeley, the Max Planck Institute for Security and Privacy, UC Santa Barbara and Arizona State University, with contributions from security researchers at Anthropic, OpenAI and Google, as a follow-on to the CyberGym benchmark (section "ExploitGym"). The preprint gives the task set: 520 userspace vulnerabilities from 161 OSS-Fuzz projects, 185 in V8 and 193 in the Linux kernel (Section 1). The model receives the source and build scripts, a proof-of-vulnerability input and description, the compiled binary, and a remote target, and must achieve unauthorized code execution and retrieve a generated flag. An attempt succeeds only if the flag is captured and a model judge finds that the intended vulnerability was used. The post's baseline runs had a two-hour limit with security mitigations turned off, each model in its developer's recommended harness (Claude Code for Claude models), and Anthropic ran the Opus 4.6 and Mythos Preview trials.
Results. Mythos Preview achieved code execution through the intended vulnerability on 157 tasks, and captured 226 flags including other paths; Opus 4.6 had 15 and 36 (section "ExploitGym", Figures 3 and 4). The figures give the other models, which the post's text does not name. GPT-5.5 had 120 intended successes and 210 flags; GPT-5.4 54 and 65; Gemini 3.1 Pro 12 and 18; Opus 4.7 7 and 9; and GLM-5.1 4 and 11. By target, Mythos Preview's 157 were 107 userspace, 38 V8 and 12 kernel, and GPT-5.5's 120 were 71, 27 and 22 (Figure 3). The post says Mythos Preview "is one of only two reported models able to frequently develop kernel exploits"; in Figure 3 the other, GPT-5.5, has more kernel successes (22 against 12).
What the preprint adds. Five conditions in the preprint bear on these numbers.
- Prompting. The post says "All models are run with identical prompts." The preprint says all agents used the same user prompt "except Claude Mythos Preview has adjusted prompting through an additional CLAUDE.md file" (Section 4).
- Judging. Two agent judges (GPT-5.5 in Codex and Opus 4.6 in Claude Code) were validated on 58 expert-labeled trajectories: Codex matched all 58 and Claude 56, both of Claude's errors being exploits of unrelated bugs labeled as intended (Section 4). The Opus 4.6 and Mythos Preview experiments used only the Claude Code judge (footnote 1).
- One attempt. Results are "a single, time-gated and cost-gated attempt per task" (Section 5, limitations). Mythos Preview and GPT-5.5 timed out on 24% and 36% of tasks, and including extended six-hour runs of Mythos Preview and Opus 4.6 raises the union of all models' solves to 239 (Section 4).
- Mitigations. When Mythos Preview's successful tasks were re-run with standard defenses turned on, 25 of its 107 userspace successes, 17 of 38 in V8 and 3 of 12 in the kernel still succeeded (Section 4, Table 5).
- Early stopping and refusals. Opus 4.7 had fewer successes than Opus 4.6; the authors' trace inspection found that it and Gemini 3.1 Pro often stopped early after judging the target vulnerability non-exploitable, and GPT-5.4 and GLM-5.1 refused 36 and 23 times on safety grounds (Section 4).
SCONE-bench: smart contracts after the knowledge cutoff
Design. SCONE-bench, built in 2025 with MATS and the Anthropic Fellows Program, asks a model to find and exploit a vulnerability in a smart contract and steal its funds in a local simulation; performance is the total simulated revenue (section "SCONE"). The updated set has 12 exploits reported after 1 January 2026, the latest knowledge cutoff of the models tested, drawn from the DeFiHackLabs dataset, with revenue converted to US dollars at the exchange rate on the day of the real exploit. Each model had eight trials per contract, scored as the best of eight, and the shaded band in Figure 5 is a 90% bootstrap confidence interval. Only Anthropic models were run.
Results. Mythos Preview exploited $35.28 million, Opus 4.7 $20.16 million, Opus 4.6 $2.05 million, Sonnet 4.6 $320,000, Opus 4.5 $71,000 and Sonnet 4.5 $67,000 (Figure 5). The post states this as $15 million, or about 75% more than the next model, and attributes the gap mainly to Mythos Preview being the only model to exploit every one of the 12 contracts. The text gives a doubling time of 0.7 months for models since Opus 4.5; Figure 5 prints "Doubling time ~0.7 months" beside a log-linear fit (R² = 0.82) whose line spans all six plotted models, Sonnet 4.5 included. The post says the December 2025 post found "a mean doubling time of 1.1 months" for models before Opus 4.5 and had remarked that "we expect the doubling trend to plateau eventually." That post prints 1.3 months, for a fit that includes Opus 4.5, in both its first archived copy (2 December 2025) and its current text, and reads "we expect this doubling trend to plateau eventually"; the 1.1 figure does not appear in it. The harness and dataset were released with the post (repository anthropics/scone-bench, initial commit e23cab9 of 18 May 2026).
What the post concludes
The post contrasts the strongest models of February 2026, which "could only barely develop exploits in simulated scenarios with most defense measures disabled," with Mythos Preview's end-to-end exploits on widely used software (Conclusion). It states, "We believe that Mythos-level models will become widely available in the next 6-12 months," calls for more public benchmarks built by domain experts, and points to the Cyber Verification Program, under which Anthropic says it can block malicious cyber use more aggressively without cutting off defenders.
Instructions and conditions described
The post prints no prompts. The conditions that shape the results are harness choices: the ExploitBench turn budget and its nudging prompts, the recommended harness per model on ExploitGym, and, per the ExploitGym preprint, an additional CLAUDE.md file for Mythos Preview (a CLAUDE.md file is a project instruction file that Claude Code reads into the model's context). ExploitGym's runs were made under OpenAI's Trusted Access for Cyber program and Anthropic's Cyber Verification Program, which the preprint says remove inference-time content filters (Section 4).
Limitations
- Self-run Claude trials. Anthropic ran all Claude trials on ExploitBench and the Mythos Preview and Opus 4.6 trials on ExploitGym. ExploitBench's authors verified the results; on ExploitGym, Anthropic's runs were judged by the Claude judge alone, the less accurate of the two in validation.
- Few attempts and no intervals. ExploitBench uses three trials per variant and ExploitGym one attempt per task; neither post nor figures give intervals for those two benchmarks.
- Prompt difference. The ExploitGym preprint's statement that Mythos Preview had an extra CLAUDE.md file is not reflected in the post.
- Mitigations off on ExploitGym. The post's ExploitGym results are with security mitigations disabled; the preprint shows large drops with them on. ExploitBench, by contrast, keeps the V8 sandbox on.
- A small, Anthropic-only SCONE set. Twelve contracts, six models from one developer, and a trend line fitted to six points.
- Comparisons the text leaves to the figures. GPT-5.5's ExploitGym results, close to Mythos Preview's in total and ahead on kernel tasks, appear only in the figures.
Research questions
- What did the CLAUDE.md file contain, and how much did it matter? Rerunning Mythos Preview on ExploitGym with and without the file, under the shared prompt, would measure its effect on the 157 successes.
- How do the ExploitGym results change with several attempts? Reporting success at several attempts per task, with intervals, would show whether the gap between Mythos Preview and GPT-5.5 is stable.
- Are Anthropic-run trials judged as strictly as the others? Scoring the Mythos Preview and Opus 4.6 trajectories with the Codex judge, or with human review, would test the single-judge condition.
- How much capability survives deployed defenses? Running every model on ExploitGym with mitigations on, not only re-running successes, would give comparable rates under realistic conditions.
- Does the SCONE trend hold on more contracts and other developers' models? A larger post-cutoff set with non-Anthropic models would test the 0.7-month doubling time.
- When does ExploitBench's ladder predict real-world exploitation? Comparing tier reached with whether exploits work against a full browser would test how far the ladder's intermediate tiers measure risk.
How it connects
- The earlier qualitative evidence. The post presents these benchmarks as quantitative support for the April 2026 case studies it links, covered in Assessing Claude Mythos Preview's cybersecurity capabilities, and for the model documented in the Claude Mythos Preview system card. It also cites its "previous findings" on Firefox (Partnering with Mozilla to improve Firefox’s security).
- SCONE-bench. The original benchmark and its 1.3-month doubling time are covered in AI agents find $4.6M in blockchain smart contract exploits.
- N-day exploits. This note connects the post to Measuring LLMs’ impact on N-day exploits, published two weeks later by overlapping authors, which measures the same models turning Firefox and Windows patches into exploits in Anthropic's own harness.
References
- Cheng, N., Lucas, K., Xiao, W., Carlini, N., & Nasr, M. (2026, May 22). Measuring LLMs’ ability to develop exploits. Anthropic Frontier Red Team. https://www.anthropic.com/research/exploit-evals. First published at https://red.anthropic.com/2026/exploit-evals/ (same date; text identical apart from figure labels). record
- Lee, S., & Brumley, D. (2026, May 15). ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents. Preprint. https://exploitbench.ai/exploitbench.pdf. Leaderboard at https://exploitbench.ai/ (the live leaderboard has since changed; Figure 1 of the post is the table at the post's date). record · 2 pages
- Wang, Z., Schiller, N., Li, H., et al. (2026). ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? arXiv:2605.11086v1. https://arxiv.org/abs/2605.11086 record
- Anthropic. (2026). SCONE-bench harness and dataset. https://github.com/anthropics/scone-bench, commit e23cab9 (18 May 2026). record