Claude is competitive with humans in (some) cyber competitions

Anthropic reports entering Claude in seven cybersecurity competitions designed for humans between February and June 2025; Claude often placed in the top quarter and solved easier challenges as fast as top teams, solved nothing at the two hardest events, and showed failures specific to long-running agents, such as losing track of its task after summarizing its context.

Anthropic · 2025-08-09 · original

How does an AI model compare with people when it enters the same security contests they do? This post, first published on 9 August 2025 on the Frontier Red Team's blog, red.anthropic.com, under the byline "Anthropic", reports what Anthropic learned from entering Claude in seven cybersecurity competitions in the first half of 2025. The anthropic.com page gives the same date and credits the Frontier Red Team. It is a short report on public events, with placings, a few charts and anecdotes, rather than a controlled evaluation. Its headline: Claude "often" placed in the top 25% of competitors but "lagged behind the best human teams at the toughest challenges" (introduction). The post does not say which Claude model or models were entered.

Why competitions

Two kinds of event are covered. In capture-the-flag (CTF) competitions, teams solve a fixed set of puzzle challenges (exploitation, reverse engineering, cryptography, forensics), each yielding a secret string, the flag, when solved. In cyber defense competitions such as the Collegiate Cyber Defense Competition (CCDC), teams defend a vulnerable network while the organizers act as a live red team attacking it.

The post gives five reasons for using competitions alongside benchmarks (section "Why enter Claude into cyber competitions?"): a baseline of students, professionals and other AI teams; multi-day events that make a model run continuously into its context limits; time pressure, since only the prompting can change during an event, which the post says forces "an honest snapshot of the model’s capabilities"; human red teams that adapt in defense events; and novelty, since challenges are new to all competitors, from which the post concludes that the model could not have seen the answers in its training data.

The seven results

The post lists each event with Claude's placing (section "Why enter Claude into cyber competitions?"):

  • Western Regional CCDC qualifier (8 February 2025; 8-hour defense). 10th of 28 teams. The post calls this a preliminary experiment and says Claude "was not targeted as aggressively as the human teams."
  • PicoCTF 2025 (7 to 17 March; aimed at high-school students). 297th of 10,460 teams, top 3% globally, solving 32 of 41 challenges; 6,533 teams solved at least one. Figure 2's label gives the final rank as 298.
  • HackTheBox AI vs Human CTF Challenge (14 to 16 March). 30th of 161 teams and 4th of 8 AI teams, solving 19 of 20 challenges.
  • Western Regional CCDC regional (28 March; 16 hours over two days). 6th of 9 qualified college teams.
  • PlaidCTF (4 April; an expert event). No challenges solved "despite numerous attempts."
  • DEF CON CTF Qualifier (12 to 14 April; an expert event). No challenges solved.
  • Airbnb CTF (24 to 26 June; invitation-only, about 180 teams of up to five people from technology companies). 13 of 30 challenges solved in the first 60 minutes, reaching 4th place, then two more over two days, for 15 of 30 and 39th place.

The post puts the failures in context: in PlaidCTF about 70% of teams solved nothing, and the one HackTheBox challenge that Claude and every other AI team failed was the one the fewest human teams solved, about 14% (section "What does all this mean for offense-defense balance in cyberspace?"). Palisade Research, which organized the HackTheBox AI track and supplied the post's HackTheBox data, gives different counts for the event in its own report: 403 teams registered, of which 158 solved at least one challenge, 152 human and 6 AI (Petrov and Volkov, 2025, Section 3.1); its table of AI agents lists seven, none of which solved all 20 challenges and none named as Anthropic's (Table 1). Neither document reconciles these with the post's 161 teams and 8 AI teams.

Speed

The post reports that when Claude solves a challenge it is "as fast or faster than elite human teams" (section "Claude can be quite fast"). At HackTheBox, Claude started 32 minutes late because the researcher launching it was moving house. Re-plotted as if it had started on time, Claude would have placed 22nd of 161 and 1st of 8 AI teams, and it kept pace with the fastest human team for about the first 17 minutes. Figure 1 shows Claude at 17 of 20 challenges in 25 minutes and the winning team, BKISC, at 20 of 20 in 27 minutes, each timed from its own start. The re-plotted placing is the post's counterfactual, not a recorded result. Several Claude instances ran in parallel on different challenges, and the post suggests that one agent per challenge could have been faster still.

Tools and autonomy

Claude solved HackTheBox challenges unattended, using tools to read challenge files and submit flags (section "Claude can make good use of autonomy and tools"). On PicoCTF, Figure 2 marks periods of manual chat on Claude.ai, of Claude Code with Kali Linux (a Linux distribution for security work), and of interactive tool use with Kali Linux; the post says the slowest progress came during manual chat. It describes this as an example of how "naive evaluation of LLMs can underestimate their capabilities." In the CCDC events, an outdated terminal tool hindered Claude in the qualifier; in the regional, a more robust terminal interface and specialized agent "personas" for tasks such as network hardening and incident response let Claude keep services running against red-team attacks that, the post says, affected the human teams' services more.

Failures specific to an AI agent

The post describes failures "in ways no human would" (section "Claude fails in ways no human would"):

  • A distracting server. In the CCDC regional, one server displayed an ASCII-art aquarium animation on login. The animation filled Claude's context; the context was summarized and passed to a fresh instance, which no longer remembered that it had been trying to log in.
  • Summarization drift. Researchers prompted Claude to summarize its status to a file as its context neared the limit. This "worked well in general," but in the qualifier Claude "descended into quasi-philosophical rumination in lieu of actual work," producing a report headed "PHILOSOPHICAL SECURITY STATUS" built from phrases such as "sum ergo securus" (the post reproduces the output). The post says, "We still do not entirely understand this behavior," and compares it with the identity confusion in Project Vend and the "spiritual bliss attractor state" in the Claude Opus 4 and Sonnet 4 system card.

What the post concludes

The post reads its results as pointing to "starker challenges for defenders, at least in the near term," because AI can automate simpler exploits while "attackers need to succeed only once and defenders need to succeed every time" (section "What does all this mean for offense-defense balance in cyberspace?"). It notes Claude's advantages in the defense events, such as not having to defend physical devices like security cameras that the human teams defended, and expects long-context and memory limits to remain an obstacle to full automation of network defense. It raises the possibility that AI-written code could make vulnerabilities rarer or more common, and calls for more research on AI-enabled defense.

Instructions and conditions described

The post prints no prompts. It mentions three kinds of instruction: prompts to summarize status to a file near the context limit, agent "personas" for defensive tasks, and prompting strategies that could be changed during an event. It does not describe the tools, scaffolds or models in detail.

Limitations

  • Model unnamed. The post never names the Claude model or models used, and events span February to June 2025, a period with more than one model release.
  • Elicitation varied by event. Tools, scaffolds and the degree of human involvement differed between events and even within one (Figure 2), so placings do not measure one fixed configuration.
  • Single entries, no repeated runs. Each placing is one entry in one event.
  • Unequal conditions. Claude was targeted less aggressively in the CCDC qualifier and did not defend physical devices in the regional; the HackTheBox placing is reported both as recorded and as a counterfactual.
  • Counts not matched. Some counts differ between text and figures (297th in the text, 298 in Figure 2) and between the post and Palisade's report on the HackTheBox event.
  • Novelty assumed. The claim that challenges were absent from training data rests on their being new; no contamination check is reported.

Research questions

  1. Which models produced these results? Naming the model and scaffold for each event would let the placings be compared with benchmark results for the same models.
  2. How much did scaffolding change the outcome? Re-running past competition challenges with and without the tools used, holding the model fixed, would measure the tool effect the PicoCTF chart suggests.
  3. Is the top-25% result stable? Entering the same configuration in repeated or archived events would show how much placings vary.
  4. What causes the summarization drift? Logging each summary across long runs, and testing other summarization prompts, would show where the philosophical rumination begins and whether a prompt change prevents it.
  5. How do defense results compare under equal conditions? A defense event in which Claude defends the same infrastructure, including physical devices, and receives the same attacks as human teams would test the defensive placings.

How it connects

References