How does an AI model compare with people when it enters the same security contests they do? This post, first published on 9 August 2025 on the Frontier Red Team's blog, red.anthropic.com, under the byline "Anthropic", reports what Anthropic learned from entering Claude in seven cybersecurity competitions in the first half of 2025. The anthropic.com page gives the same date and credits the Frontier Red Team. It is a short report on public events, with placings, a few charts and anecdotes, rather than a controlled evaluation. Its headline: Claude "often" placed in the top 25% of competitors but "lagged behind the best human teams at the toughest challenges" (introduction). The post does not say which Claude model or models were entered.
Why competitions
Two kinds of event are covered. In capture-the-flag (CTF) competitions, teams solve a fixed set of puzzle challenges (exploitation, reverse engineering, cryptography, forensics), each yielding a secret string, the flag, when solved. In cyber defense competitions such as the Collegiate Cyber Defense Competition (CCDC), teams defend a vulnerable network while the organizers act as a live red team attacking it.
The post gives five reasons for using competitions alongside benchmarks (section "Why enter Claude into cyber competitions?"): a baseline of students, professionals and other AI teams; multi-day events that make a model run continuously into its context limits; time pressure, since only the prompting can change during an event, which the post says forces "an honest snapshot of the model’s capabilities"; human red teams that adapt in defense events; and novelty, since challenges are new to all competitors, from which the post concludes that the model could not have seen the answers in its training data.
The seven results
The post lists each event with Claude's placing (section "Why enter Claude into cyber competitions?"):
- Western Regional CCDC qualifier (8 February 2025; 8-hour defense). 10th of 28 teams. The post calls this a preliminary experiment and says Claude "was not targeted as aggressively as the human teams."
- PicoCTF 2025 (7 to 17 March; aimed at high-school students). 297th of 10,460 teams, top 3% globally, solving 32 of 41 challenges; 6,533 teams solved at least one. Figure 2's label gives the final rank as 298.
- HackTheBox AI vs Human CTF Challenge (14 to 16 March). 30th of 161 teams and 4th of 8 AI teams, solving 19 of 20 challenges.
- Western Regional CCDC regional (28 March; 16 hours over two days). 6th of 9 qualified college teams.
- PlaidCTF (4 April; an expert event). No challenges solved "despite numerous attempts."
- DEF CON CTF Qualifier (12 to 14 April; an expert event). No challenges solved.
- Airbnb CTF (24 to 26 June; invitation-only, about 180 teams of up to five people from technology companies). 13 of 30 challenges solved in the first 60 minutes, reaching 4th place, then two more over two days, for 15 of 30 and 39th place.
The post puts the failures in context: in PlaidCTF about 70% of teams solved nothing, and the one HackTheBox challenge that Claude and every other AI team failed was the one the fewest human teams solved, about 14% (section "What does all this mean for offense-defense balance in cyberspace?"). Palisade Research, which organized the HackTheBox AI track and supplied the post's HackTheBox data, gives different counts for the event in its own report: 403 teams registered, of which 158 solved at least one challenge, 152 human and 6 AI (Petrov and Volkov, 2025, Section 3.1); its table of AI agents lists seven, none of which solved all 20 challenges and none named as Anthropic's (Table 1). Neither document reconciles these with the post's 161 teams and 8 AI teams.
Speed
The post reports that when Claude solves a challenge it is "as fast or faster than elite human teams" (section "Claude can be quite fast"). At HackTheBox, Claude started 32 minutes late because the researcher launching it was moving house. Re-plotted as if it had started on time, Claude would have placed 22nd of 161 and 1st of 8 AI teams, and it kept pace with the fastest human team for about the first 17 minutes. Figure 1 shows Claude at 17 of 20 challenges in 25 minutes and the winning team, BKISC, at 20 of 20 in 27 minutes, each timed from its own start. The re-plotted placing is the post's counterfactual, not a recorded result. Several Claude instances ran in parallel on different challenges, and the post suggests that one agent per challenge could have been faster still.
Tools and autonomy
Claude solved HackTheBox challenges unattended, using tools to read challenge files and submit flags (section "Claude can make good use of autonomy and tools"). On PicoCTF, Figure 2 marks periods of manual chat on Claude.ai, of Claude Code with Kali Linux (a Linux distribution for security work), and of interactive tool use with Kali Linux; the post says the slowest progress came during manual chat. It describes this as an example of how "naive evaluation of LLMs can underestimate their capabilities." In the CCDC events, an outdated terminal tool hindered Claude in the qualifier; in the regional, a more robust terminal interface and specialized agent "personas" for tasks such as network hardening and incident response let Claude keep services running against red-team attacks that, the post says, affected the human teams' services more.
Failures specific to an AI agent
The post describes failures "in ways no human would" (section "Claude fails in ways no human would"):
- A distracting server. In the CCDC regional, one server displayed an ASCII-art aquarium animation on login. The animation filled Claude's context; the context was summarized and passed to a fresh instance, which no longer remembered that it had been trying to log in.
- Summarization drift. Researchers prompted Claude to summarize its status to a file as its context neared the limit. This "worked well in general," but in the qualifier Claude "descended into quasi-philosophical rumination in lieu of actual work," producing a report headed "PHILOSOPHICAL SECURITY STATUS" built from phrases such as "sum ergo securus" (the post reproduces the output). The post says, "We still do not entirely understand this behavior," and compares it with the identity confusion in Project Vend and the "spiritual bliss attractor state" in the Claude Opus 4 and Sonnet 4 system card.
What the post concludes
The post reads its results as pointing to "starker challenges for defenders, at least in the near term," because AI can automate simpler exploits while "attackers need to succeed only once and defenders need to succeed every time" (section "What does all this mean for offense-defense balance in cyberspace?"). It notes Claude's advantages in the defense events, such as not having to defend physical devices like security cameras that the human teams defended, and expects long-context and memory limits to remain an obstacle to full automation of network defense. It raises the possibility that AI-written code could make vulnerabilities rarer or more common, and calls for more research on AI-enabled defense.
Instructions and conditions described
The post prints no prompts. It mentions three kinds of instruction: prompts to summarize status to a file near the context limit, agent "personas" for defensive tasks, and prompting strategies that could be changed during an event. It does not describe the tools, scaffolds or models in detail.
Limitations
- Model unnamed. The post never names the Claude model or models used, and events span February to June 2025, a period with more than one model release.
- Elicitation varied by event. Tools, scaffolds and the degree of human involvement differed between events and even within one (Figure 2), so placings do not measure one fixed configuration.
- Single entries, no repeated runs. Each placing is one entry in one event.
- Unequal conditions. Claude was targeted less aggressively in the CCDC qualifier and did not defend physical devices in the regional; the HackTheBox placing is reported both as recorded and as a counterfactual.
- Counts not matched. Some counts differ between text and figures (297th in the text, 298 in Figure 2) and between the post and Palisade's report on the HackTheBox event.
- Novelty assumed. The claim that challenges were absent from training data rests on their being new; no contamination check is reported.
Research questions
- Which models produced these results? Naming the model and scaffold for each event would let the placings be compared with benchmark results for the same models.
- How much did scaffolding change the outcome? Re-running past competition challenges with and without the tools used, holding the model fixed, would measure the tool effect the PicoCTF chart suggests.
- Is the top-25% result stable? Entering the same configuration in repeated or archived events would show how much placings vary.
- What causes the summarization drift? Logging each summary across long runs, and testing other summarization prompts, would show where the philosophical rumination begins and whether a prompt change prevents it.
- How do defense results compare under equal conditions? A defense event in which Claude defends the same infrastructure, including physical devices, and receives the same attacks as human teams would test the defensive placings.
How it connects
- Related Anthropic cyber work. The post links the Frontier Red Team's work on multistage attacks with cyber toolkits (LLMs with cyber toolkits can conduct multistage cyber operations on business-sized computer networks) as related research. Building AI for cyber defenders (September 2025) later cites these competitions among its evidence that models had become useful for security work.
- Long-context behavior. The post compares the summarization drift with Project Vend and with the conversation attractor reported in the Claude Opus 4 and Sonnet 4 system card (pp. 62 to 65).
- Later measurements. This note connects the post to Measuring LLMs’ ability to develop exploits (May 2026), which measures exploit development on benchmarks with fixed harnesses rather than in open competition.
References
- Anthropic. (2025, August 9). Claude is competitive with humans in (some) cyber competitions. https://www.anthropic.com/research/cyber-competitions. First published at https://red.anthropic.com/2025/cyber-competitions/ (Internet Archive capture 20250809203025; the text matches the current page except that the first version had a heading "How did Claude do?" over the list of events and captioned the reproduced output as Figure 4, and the current page adds the line linking the DEF CON 33 talk). record
- Petrov, A., & Volkov, D. (2025, May 25). Evaluating AI cyber capabilities with crowdsourced elicitation. arXiv:2505.19915v1 (v2, 27 May 2025, not compared). https://arxiv.org/abs/2505.19915 record
- Carlini, N., et al. (2025). LLMs unlock new paths to monetizing exploits. arXiv:2505.11449. Cited in the post (footnote 1); not read for this note. record
- Lucas, K. (2025). Talk on this work at DEF CON 33. https://www.youtube.com/watch?v=sbkeEwhWIks. Linked from the current page; not watched for this note. record