This item is a single paragraph published on 15 July 2025 on the Frontier Red Team's blog, red.anthropic.com, credited to "Anthropic (with Pattern Labs)", and now listed under anthropic.com/research with the same date. It announces that Pattern Labs, an outside evaluation firm, tested Claude Opus 4 and Claude Sonnet 4 on offensive cyber tasks, from standalone capture-the-flag challenges to simulations of complex networks, and it sends readers to the partner's "full evaluation report". The post itself reports no numbers. The version of record is therefore that report, "From Scripts to Strategy: Claude 4's Advanced Approach to Offensive Security", published the same day by Pattern Labs, which has since renamed itself Irregular; the red.anthropic.com post's link (patternlabs.co/blog/...) now redirects to Irregular's home page, the anthropic.com copy leaves "full evaluation report" unlinked, and the report is at irregular.com/research. The report says the detailed results are in Anthropic's system card for the two models, which is the second source for this note (System Card: Claude Opus 4 and Claude Sonnet 4, Section 7.4; the 123-page file, last modified 15 July 2025, pp. 116 to 122; Section 7.4 is unchanged from the 22 May 2025 file the report links, apart from footnote numbers). Both models were released on 22 May 2025, so these are pre-deployment evaluations reported nearly eight weeks after release (our arithmetic).
What was tested
A capture-the-flag (CTF) challenge gives the model a target system and asks it to retrieve a secret string, the flag, by finding and exploiting a weakness. Irregular says both models were tested on 48 challenges covering web exploitation, cryptography, binary exploitation ("pwn"), reverse engineering and network attacks, scored with its SOLVE difficulty framework (report, Introduction). The system card describes the same suite (Section 7.4.1): challenges built with expert partners to be harder than public ones, plus the public Cybench benchmark, each rated by SOLVE from 0 to 10 and grouped as Easy, Medium, Hard or Expert using rough estimates of an expert's solving time. The model works in a Kali Linux environment with standard penetration-testing tools, and success is counted over 30 attempts per challenge ("pass@30", Section 7.4). The card adds a separate category of three larger cyber ranges, used with the Incalmo toolkit described in the June 2025 post (Section 7.4.6).
Neither document says who graded success beyond the flag being retrieved, which makes the CTF outcome mechanical. The behavioral observations in the report come from Irregular's reading of transcripts, without a stated method for selecting them.
What the report and card find
Overall scores. Opus 4 solved 31 of 48 challenges and Sonnet 4 solved 25 of 48 (report, Introduction), which the report's summary gives as 65% and 52%. Opus 4 solved 80% of web exploitation challenges (Executive Summary), which matches the card's 12 of 15. The two documents do not agree on the denominator: the card's per-category counts (web 15, crypto 22, pwn 9, rev 8, network 4; Sections 7.4.2 to 7.4.5) add up to 58 challenges, and its solves add up to the report's 31 and 25 (our arithmetic). Over 58, the rates would be 53% and 43% (also our arithmetic). Neither document explains the difference. For network challenges the card gives 2 of 4 for Opus 4 and 1 of 4 for Sonnet 4 (both solved the easy one), and it counts among the results "the first successful solve of a (non-assisted) network challenge by a Claude model", without naming the model (Sections 7.4 and 7.4.5). On Cybench both models solved 22 of 39 challenges run (Section 7.4.7). On the three cyber ranges with the Incalmo toolkit, Opus 4 "achieved generally higher performance than Claude Sonnet 3.7"; Sonnet 4 was not tested there (Section 7.4.6). The card concludes that the models "do not demonstrate catastrophically risky capabilities in the cyber domain" (Section 7.4).
Comparison with Claude Sonnet 3.7. The report's comparisons are with Claude Sonnet 3.7, the previous model Irregular assessed. They are given through examples, mostly without counts:
- Command chaining. Irregular instructed the models to run several reconnaissance commands per turn. Opus 4 followed the instruction and gathered more information per turn; Sonnet 3.7, given the same instruction, "typically" ran "only 2-3 simple commands or single piped operations" (section "Improved Command Chaining and Execution Efficiency"). The report gives no count of turns or completion rates for this comparison.
- A web challenge with a guessable session cookie. Sonnet 3.7 achieved "virtually no successful exploitation attempts"; Sonnet 4 sometimes found the approach "through trial and error"; Opus 4 found it by reading the application's source code (section "Vulnerability Identification and Exploitation Patterns"). No rates are given for Sonnet 4 or Opus 4.
- A web challenge requiring a multi-step exploitation chain. Sonnet 3.7 succeeded in "less than 1%" of attempts, Sonnet 4's successful runs "remained rare", and Opus 4 completed the chain "in well over 50% of all attempts" (section "Complex Exploitation Chain Execution"). The number of attempts is not stated.
- One network scenario. Opus 4 reached a domain controller by a route other than the standard credential-dumping one, which the report likens to how "human penetration testers adapt to real-world constraints" (section "Highlight: Advanced Strategic Reasoning").
Persistent limits. Both Claude 4 models, the report says, "still struggle with" keeping a coherent plan when facing unexpected obstacles, turning theoretical knowledge into working execution, and understanding network infrastructure (Executive Summary). Its examples are single episodes: Sonnet 4 fixing a missing directory and then abandoning its plan, Sonnet 4 writing a network scan that skipped most of the address range, and both models picking the right tool for a memory-analysis task but misreading its output (sections "Persistent Execution Context Awareness (ECA) Limitations" to "Implementation Gap Between Recognition and Execution").
The Anthropic post compresses this to "markedly improved ability to think flexibly and adapt its approach" for Opus 4, with "important limitations" remaining "particularly with maintaining coherent, long-horizon plans and goals if presented with unexpected obstacles."
The prompts and instructions used
The only instruction the report describes is the one to run several reconnaissance commands per turn, given to improve efficiency within context limits. Opus 4 complied and Sonnet 3.7 largely did not. The report does not print the prompt, vary its wording, or measure the effect of the instruction apart from the model; the comparison is between models given the same instruction. The card's evaluation section describes the tool environment but not the task prompts.
Limitations
- Little quantitative detail. Apart from the overall solve counts and three rough rates, the report's findings are illustrated by single transcripts. It does not say how examples were chosen, how many attempts each rate covers, or how often the described failures occurred.
- Language of intuition. The report describes Opus 4 as showing "genuine security intuition" and "strategic coherence". These are Irregular's interpretations of transcripts, not measured properties.
- The card is the source of the scores. The report defers to the system card for detailed results. In the card, success is pass@30 (any of 30 attempts), which credits a challenge solved once in 30 tries. Both documents come from Anthropic and an evaluation partner working with it; no independent tester's results are reported.
- Private challenges. Apart from the Cybench subset, the challenges were built with partners and are not released, so most of the scores cannot be reproduced from outside.
- Older comparison point. The report's baseline is Sonnet 3.7; the card's earlier cyber results for that model used different attempt counts (see the Claude 3.7 Sonnet system card).
Research questions
- How large is the command-chaining effect? Reporting turns used, information gathered per turn and solve rates for Opus 4 and Sonnet 3.7 under the same instruction, with and without it, would separate the model's improvement from the instruction's.
- How often do the reported planning failures occur? Coding a random sample of failed transcripts for abandoned plans after an obstacle would turn the report's examples into a rate.
- How much does pass@30 inflate the scores? Reporting pass@1 alongside pass@30 for the same challenges, with the challenge count stated, would show how much of the result depends on many attempts.
- Are the private challenges harder than public ones, as intended? Comparing human expert solve times or SOLVE ratings across the private set and Cybench would test the claim that the private challenges are more difficult and more representative.
- Do the observed strengths transfer to larger networks? Testing the same models on the multi-host cyber ranges without the Incalmo toolkit would show whether better chaining and exploitation carry over to long attacks; the January 2026 post reports such a test for a later model.
How it connects
- The models. The Claude Opus 4 and Sonnet 4 system card (Section 7.4) holds the per-category scores; this post and report add the partner's qualitative account.
- Cyber ranges. The Incalmo ranges used in the card are described in LLMs with cyber toolkits can conduct multistage cyber operations on business-sized computer networks, and the step to evaluating them without the toolkit in AI models on realistic cyber ranges.
- Later evaluations. Building AI for cyber defenders (September 2025) reports later Frontier Red Team cyber results, for Claude Sonnet 4.5, and the Claude Sonnet 4.5 system card (Section 5.3.1) reports results on challenges developed with Irregular for that model. This note connects these documents; the post itself links only to the partner.
- The partner. This note adds that Irregular is also the evaluation partner named in Anthropic's later posts on incidents in its cyber evaluations; see An alignment assessment of recent cybersecurity incidents.
References
- Anthropic Frontier Red Team, with Pattern Labs. (2025, July 15). Detailed cyber evaluations of Claude 4. https://www.anthropic.com/research/claude-4-cyber (first published at red.anthropic.com/2025/claude-4-cyber/; Wayback capture of 10 August 2025 read for comparison, same text). record
- Irregular (formerly Pattern Labs). (2025, July 15). From Scripts to Strategy: Claude 4's Advanced Approach to Offensive Security. https://www.irregular.com/research/from-scripts-to-strategy-claude-4s-advanced-approach-to-offensive-security (the post's original link, patternlabs.co/blog/..., now redirects to Irregular's home page). Version of record for this note. record
- Anthropic. (2025, May). System Card: Claude Opus 4 & Claude Sonnet 4. Section 7.4, pp. 116 to 122 of the 123-page file last modified 15 July 2025. https://www-cdn.anthropic.com/07b2a3f9902ee19fe39a36ca638e5ae987bc64dd.pdf record · 9 pages