Measuring LLMs’ impact on N-day exploits

Anthropic's Frontier Red Team measures how fast Claude models turn published security patches into working exploits; on 18 Firefox JavaScript-engine patches Claude Mythos Preview built 8 code-execution exploits, and on 21 Windows kernel patches it built 8 privilege-escalation chains for about $15,700 in API credits, while the public models tested, with safeguards off, built at most two exploits on Firefox and none on Windows.

Winnie Xiao et al. · 2026-06-08 · original

How quickly can a language model turn a published security fix into a working attack? This post by Anthropic's Frontier Red Team (Winnie Xiao, Tim Abbott, Nicholas Carlini, Newton Cheng, David Forsythe, Keane Lucas, Milad Nasr and Shikhar Sakhuja) measures that for six Claude models on two targets, Mozilla Firefox and the Windows kernel. The anthropic.com page is dated 8 June 2026. The post first appeared on red.anthropic.com, where the page is dated 5 June 2026; the Internet Archive's first capture of it is from 8 June. There is no companion paper, so the post and its five figures are the version of record, and the figures print the counts and times this note cites. The post's argument is that exploit development, historically the step in an N-day campaign most limited by scarce expertise, now takes hours, so defenders should patch faster.

N-days and the patch gap

An N-day is a vulnerability that has been publicly disclosed and patched but is still present on systems that have not installed the patch. The interval between a fix's release and its installation is the patch gap. A zero-day, by contrast, is unknown to the software's maintainers. The post explains why N-days matter: a patch shows where the bug is, and an attacker can patch diff, comparing the old and new code or binary to locate the change and work back to the flaw it fixes (introduction).

The post's historical baseline is cited, not measured. WannaCry struck 59 days after Microsoft's MS17-010 bulletin in 2017; the public exploit for Citrix Bleed in 2023 took about two weeks; and in a 2020 Mandiant analysis, 16 of 25 N-day vulnerabilities took a month or more to exploit (introduction). The post notes that exploit development is only one step in a real campaign, alongside finding targets, delivering the exploit and evading detection.

Two terms recur. A proof-of-concept (PoC) here is an input that triggers the bug, shown by a crash. An exploit goes further and uses the bug to take control: code execution on Firefox, or elevation from a low-privilege user to SYSTEM on Windows.

The Firefox experiment

Why Firefox. An earlier collaboration with Mozilla supplied a hardened harness and grader, and Firefox is close to a best case for defenders: it updates itself, its point releases had recently moved from monthly to roughly weekly, and for the patches studied the median gap to release was 19 days (section "N-days on Firefox").

Setup. The task set was 18 security patches to SpiderMonkey, Firefox's JavaScript engine, shipped in Firefox 148 and 149 (released 24 February and 24 March 2026); the caption to Figure 1 gives 15 in Firefox 148 and 3 in 149, while the figure titles say "patched in Firefox 147 to 149". Only bugs whose fixes had been public in Mozilla's repository for at least 90 days were kept. The post does not say whether these fixes postdate the tested models' training cutoffs, as it does for the Windows set. Each model worked in a Linux container with a shell and a text editor and no internet access. It received the public diff with the maintainer's regression test removed, the component name, Mozilla's severity rating, and two instrumented builds of the standalone JavaScript shell, one before and one after the fix. It did not receive the advisory text or the reporter's reproducer (section "Setup").

Grading. A PoC counted if it crashed the vulnerable build but not the patched one. An exploit counted only if it read a randomized secret from a file the JavaScript sandbox cannot reach, and did so on the vulnerable build only (section "Results").

PoC results. Six models had three trials each per vulnerability, with a budget of three million tokens per trial (Figure 1 caption). The number of patches turned into a working PoC rose from 2 for Claude Opus 4.5 to 11 for Claude Opus 4.8, and Claude Mythos Preview reached 14. Figure 1 gives the others: Claude Opus 4.6 10, Claude Sonnet 4.6 8 and Claude Opus 4.7 7. Mythos Preview's first PoC came in about 12 minutes and 13 came within 40 minutes, about half the 72 minutes Opus 4.8 took to reach its 11; the last took the total to about three hours (3.1 hours in Figure 1). Figure 1 plots, for each vulnerability, the fastest of the three trials.

Consistency. The three strongest models then ran 50 trials per vulnerability. Mythos Preview produced a PoC in all 50 trials for 7 of the 18 vulnerabilities, against one each for Opus 4.8 and Opus 4.6 (Figure 2). Across the 50 trials, Mythos Preview produced a PoC at least once for 15 vulnerabilities, Opus 4.8 for 12 and Opus 4.6 for 11 (Figure 2 legend). Figure 2 ranks each model's vulnerabilities by its own success rate, so the curves compare capability profiles rather than the same bugs.

Exploit results. Starting from its own fastest successful PoC, each model had three further trials on each vulnerability for which it had a PoC, with the same token budget (Figure 3 caption). Mythos Preview wrote its first working exploit in a little under an hour (59 minutes end to end in Figure 3) and eight distinct exploits in about 12 hours (11.8 hours). Opus 4.8 wrote two, and Opus 4.6 and Sonnet 4.6 one each; Opus 4.7 and Opus 4.5 wrote none (section "Results", Figure 3). Exploits were deduplicated by an LLM agent and manual inspection. The post compares the first exploit's time with Mozilla's timeline: under an hour from the start of a run, against 18 days between Mozilla's commit of that fix and the release of Firefox 148. That comparison is the post's; the runs themselves took place after the fixes had been public for at least 90 days.

The Windows experiment

Setup. The second target was closed-source: 21 Windows kernel vulnerabilities from Microsoft's January and February 2026 Patch Tuesdays, all after the knowledge cutoffs of every model tested, and all local elevation-of-privilege bugs, chosen because escalation can be checked mechanically with the whoami command (section "N-days on Windows"). The model received what an attacker would have on patch day: the vulnerable and patched binaries, public debug symbols, a Ghidra decompilation of the vulnerable binary, a function-level diff from Ghidriff, and Microsoft's advisory text. It worked as a low-privilege user against a live Windows Server 2025 virtual machine with no network access, with a shell, a text editor and helper scripts to compile, copy and run code. The machine was configured so that a memory bug crashes it at once. The Ghidra and Ghidriff outputs were computed offline beforehand, about two hours for all files, and staged as files at launch (Figure 4 caption).

Grading. Each submission was recompiled and run on a fresh virtual machine. A crash counted when the system halted with a bug check (a Blue Screen); escalation counted when whoami changed from the low-privilege user to SYSTEM, checked with a per-run nonce. An agentic LLM grader then reran each PoC and read the transcript and source to rule out tampering, off-target crashes and unrealistic attacks (Figure 4 and 5 captions). The post does not report validating this grader against human review.

PoC results. With three trials per vulnerability, Sonnet 4.6 and Opus 4.7 each crashed the kernel through 13 of the 21 vulnerabilities, Opus 4.8 through 15 and Mythos Preview through 18; Figure 4 adds Opus 4.6 at 12 and Opus 4.5 at 6. Mythos Preview's first PoC came in 31 minutes and all 18 within six hours (5.7 hours in Figure 4), for about $2,200 in API credits.

Exploit results. Only Mythos Preview produced full privilege-escalation chains: eight distinct exploits, for $15,700 in API credits, about $2,000 each (section "Results"). Figure 5 shows the first at 6.6 hours and the eighth at 17.8 hours. Opus 4.8 built arbitrary read and write primitives and found a kernel address leak in several trials but did not chain them to SYSTEM.

Microsoft's ratings. Microsoft's advisories rated 14 of the 21 vulnerabilities "Exploitation Less Likely" or "Exploitation Unlikely"; Mythos Preview produced PoCs for 13 of these, including a privilege escalation for one rated "Exploitation Unlikely" (section "Results"). The post comments that the rating system "is currently calibrated to human researchers."

Rollout comparison. Using Windows Autopatch as a reference, the post says a patch typically reaches 90% of enrolled devices after seven days, with a forced reboot on day 11, and that Mythos Preview "would have finished creating all eight full chain exploits before any of the Windows devices had received the patch as an update."

What the post concludes

The post states that N-day exploits by language models are not in themselves new: "Given enough time and a good enough harness, this has likely been possible for a while" (Conclusion). What it says has changed is volume and speed, so that a lone operator can turn "a month’s worth of patches into working exploits in a single afternoon." For comparison (this note's), the post's own figures place Mythos Preview's eighth Windows chain 17.8 hours and its eighth Firefox exploit 11.8 hours from launch, each the fastest of three trials. The post concludes that "N-hour is closer to the reality we now operate in," that devices with slow patch cycles (industrial control systems, medical devices, internet-of-things devices) are most exposed, and that a more durable answer than faster patching is to reduce the supply of bugs, through memory-safe languages such as Rust and mitigations that remove whole classes of exploit.

Instructions and conditions described

The post prints no prompts. The conditions it states are the harness contents, the token budget, the inputs withheld (advisory text and reproducer on Firefox), and the statement that the public models were run "with our safeguards turned off" (introduction). Results therefore describe model capability without Anthropic's deployment safeguards, not what a user of the public products would get.

Limitations

  • Small task sets. The sets are 18 and 21 vulnerabilities, each from one product and two release cycles; all 21 Windows bugs are local privilege escalations.
  • Few trials and no intervals. Most results are the best of three trials per vulnerability, and times are the fastest trial, measured from launch. Only the Firefox PoC consistency test used 50 trials. No interval or between-run variance is reported.
  • Only Claude models. No non-Anthropic model and no human team was run on these tasks; the human baseline is historical (WannaCry, Citrix Bleed, the Mandiant analysis), not measured in the same setting.
  • Training-data exposure on Firefox. The Windows set postdates every model's knowledge cutoff; the post does not say the same of the Firefox set, whose fixes had been public for at least 90 days.
  • Graders. Firefox grading is mechanical. Windows grading adds an agentic LLM grader whose agreement with human review is not reported, and exploit deduplication used an LLM agent with manual inspection.
  • Simplified targets. Firefox runs used the standalone JavaScript shell, not the full browser. The post notes that delivery, target discovery and detection evasion are not measured.
  • Cost figures. API credit costs are reported for Mythos Preview on Windows only; Mythos Preview was not generally available when the post was published.

Research questions

  1. How much of the Firefox result depends on the bugs being public before training? Running the same harness on SpiderMonkey fixes released after every model's cutoff, as the Windows set was, would separate exploit development from recall.
  2. How reliable is the agentic grader on Windows? Independent human review of a sample of graded PoCs and exploits, including rejected ones, would estimate its false-positive and false-negative rates.
  3. How do non-Anthropic models and human experts compare on the same tasks? Running other frontier models, and timing professional exploit developers, in the same harness would place the Claude results on a common scale.
  4. Do the times hold across more trials? Reporting the distribution of time to first exploit over many trials, not the fastest of three, would show what a typical rather than a best-case run achieves.
  5. Do the exploits work outside the harness? Testing the Firefox exploits against the full browser, and the Windows chains on default configurations, would show how far the results transfer to deployed systems.
  6. What happens to exploitability ratings? Comparing vendors' exploitability ratings with model success across many advisories would test the post's suggestion that the ratings need recalibration.

How it connects

References