This post asks whether language models that were not trained for cybersecurity can carry out an attack that spans many machines on a network: get in from outside, move from host to host, and take data out. It appeared on 13 June 2025 on the Frontier Red Team's blog, red.anthropic.com, credited to "Anthropic (with Carnegie Mellon University's CyLab)", and now sits under anthropic.com/research with the same date and wording. The post is a one-page summary. Its numbers come from a paper by Brian Singer and co-authors at Carnegie Mellon University, with Keane Lucas of Anthropic, which the post cites, without a version number, by its first arXiv title, "On the Feasibility of Using LLMs to Execute Multistage Network Attacks". Two arXiv versions carry that title: v1 (27 January 2025) and v2 (6 March 2025). They report the same results, figures and section numbers; v2 drops v1's footnote saying that Incalmo could not yet be released publicly because disclosure was in progress, and its plan to notify major model providers before publication. v2, the latest version that matches the post, is the version of record for this note, and the locations below hold for both. The post's model list and figures match these versions, not the later ones (see "Later versions of the paper"). At the time, published tests of models on security mostly used capture-the-flag (CTF) puzzles on a single machine, and the post presents these networks as "more realistic and sophisticated" than those.
What the research does
The networks. The paper builds ten emulated networks of 25 to 50 hosts (Section 2.2, Table 1, Appendix A). Two copy real incidents from public reports: a replica of the network in the Equifax data breach, where the goal is to take data from 48 databases, and an environment inspired by the Colonial Pipeline attack, with an operational-technology network whose 5 critical actuator hosts are the goal (Appendix A). The other eight use topologies common in enterprises or used in earlier research (chains, stars, a "dumbbell" of web servers and databases, two tree-shaped enterprise layouts). All vulnerabilities are known ones, such as a public web-framework flaw and privilege-escalation bugs with published identifiers; none requires discovering a new vulnerability.
Incalmo. The toolkit sits between the model and the network. Instead of typing shell commands, the model writes short Python tasks that call five high-level actions modeled on the MITRE ATT&CK framework: scan a network, move laterally to a host, escalate privileges, find information on a host, and exfiltrate data (Section 4). Incalmo's own code, built on MITRE's open-source Caldera emulation framework and involving no language model, turns each action into the low-level commands, using a small hand-built library that maps known vulnerabilities to exploits (Section 6). Two services give the model structure: an environment-state service that records what has been discovered, and an attack-graph service that suggests which hosts and vulnerabilities could be used next. An attack graph here is the paper's map of states an attacker can reach (for example, "has access to the web server") and the actions that connect them; the Equifax replica's graph has 246 states (Figure 3).
Baselines. Without Incalmo, three models (Claude Sonnet 3.5, GPT-4o and Gemini 1.5 Pro) ran with a shell on a Kali Linux attack machine, 5 trials per network, and the human-in-the-loop tool PentestGPT was run by hand, 3 trials per network, executing only the concrete commands it gave (Sections 2.2 and 2.3; the paper does not name the model behind PentestGPT). OpenAI's o1 could not be tested because its public API blocked attack tasks (Section 2.2).
Runs and scoring. With Incalmo, six models were tested: Sonnet 3.5, Haiku 3.5, GPT-4o, GPT-4o mini, Gemini 1.5 Pro and Gemini 1.5 Flash, 5 trials per network, each limited to 75 minutes (Section 7.1). Full success means reaching every goal (all 48 databases, for example); partial success means reaching at least one. Results are reported as the best of the 5 trials, which the authors justify on the ground that "attackers can rerun the LLM on the network multiple times" (Section 7.2). The success criteria are mechanical (data taken, hosts reached), so no model or human grades the outcomes; the attack-graph coverage measure was built by the authors from a hand-written reference attack for each network (Section 3).
What it finds
- Without the toolkit, almost nothing. No model fully succeeded in any network. Sonnet 3.5 took 1 of 25 files in the 4-layer chain network in a single trial; GPT-4o, Gemini 1.5 Pro and PentestGPT reached no goal anywhere (Section 2.3, Figure 2).
- Why models failed. For two networks the authors labeled every shell command. Between 28% and 81% of commands were irrelevant to any path through the attack graph, and 9% to 32% of the relevant ones were wrongly implemented, for example a scan with parameters that hid the vulnerability (Section 3.3). Of the 235 relevant and correct commands, 229 were scans that found hosts.
- With the toolkit. Models fully succeeded in 5 of 10 networks and partly in 9 of 10 (Finding 1.A, Section 7.2, Figure 9). In the Equifax replica all six models took some data and four took all of it. Enterprise B, which needed both an outside scan and an inside scan, defeated every model (Section 8).
- Coverage. The best run per model reached 48% to 100% of attack-graph states with Incalmo against 1% to 30% without (Finding 1.B, Figure 10). The authors note one exception: Gemini 1.5 Flash did not reach more states with Incalmo in every network (Figure 10 caption).
- Which part matters. Removing the high-level actions but keeping the services left all six models unable to reach any goal; removing the services but keeping the actions cost 1 to 5 networks per model (Findings 2.A and 2.B, Section 7.3, Figures 11 and 12). Small models with Incalmo did better than large models without it in 9 of 10 networks (Finding 3, Figure 13), which the authors read as abstraction mattering more than model size in this setting.
- Reliability. Taking the best of five runs flatters consistency. Only Haiku 3.5 in the Colonial Pipeline network fully succeeded in all 5 trials, and Gemini 1.5 Pro fully succeeded in 4 of 5 Equifax trials; Haiku 3.5 and Sonnet 3.5 partly succeeded in all 5 trials in 5 of the 10 networks (Section 7.3 and Appendix B, Figure 14).
The post restates these results in plain terms: "fully compromising 5 out of 10 test networks and partially compromising 4 others, compared to almost complete failure without the Incalmo cyber toolkit" (summary bullets). It adds that all but one model at least partly succeeded on the Colonial Pipeline network (the exception in its chart is Gemini 1.5 Flash), and it stresses that the models were "not fine-tuned for cybersecurity".
What the post adds about scaling
The post offers one example of general scaling: Sonnet 3.5 "outperforms a smaller model (Claude Haiku 3.5)" when neither has Incalmo. This note could not locate that comparison in v1 or v2. Figure 2, which reports goal success without Incalmo, covers only the three larger models; Figure 10 shows Haiku 3.5's attack-graph coverage without Incalmo, and there the six models' bars are close to one another (read from the figure, which prints no values). The comparison may rest on Sonnet 3.5's single exfiltrated file.
The post's wider claims are framed as possibilities: these results "show how LLMs could lower the barriers to conducting complex cyber attacks", and scaling, better toolkits and cyber fine-tuning are "all vectors for these capabilities to develop rapidly." It also points to the defensive use the paper argues for, automated penetration testing.
The prompts and instructions used
The post says prompting "was limited to introducing Incalmo, the scenario, and the goal". The paper prints the opening of that prompt, which casts the model as "a world-class white hat hacker testing a network for a cybersecurity exercise" and asks it to "obtain access to critical hosts and exfiltrate all critical data" (Section 5). The rest teaches the toolkit's interface (how to mark a query or an action) and gives the network's starting address range. The paper calls this goal "environment-agnostic" and describes the onboarding prompt as "LLM-agnostic" (Sections 4 and 5). It does not vary its wording, so it offers no evidence on how wording affected results; its claim is about the interface (high-level actions against raw shell commands), not the instructions.
Limitations
The post and paper state these themselves:
- No defenders. "A limitation of the research setup is the lack of active defenses on the simulated networks, making them easier to compromise" (post). The paper also leaves detection and evasion to future work (Section 8).
- Known vulnerabilities and fitted tooling. The attacks "still relied on known vulnerabilities, not the discovery and exploitation of novel vulnerabilities", and "some tooling in Incalmo was built specifically with these research scenarios in mind; new tools would need to be added to threaten real-world networks" (post). The exploit library was written by hand for these networks (Section 6).
- Constructed networks. The authors call the environments a "best effort" at realism from public sources, since real enterprise networks are rarely documented in public (Section 8).
Further points a reader should weigh:
- Best-of-five scoring. Headline success is the best of five runs. Full success on all five runs occurred once.
- Small samples, no intervals. Each model-network pair has 5 trials (3 for PentestGPT), and no confidence intervals are reported.
- Who did the hard steps. With Incalmo, the model plans and Incalmo's hand-built agents execute; the exploits themselves come from the library. The success measures what a model can orchestrate given those agents, which is the authors' stated point. In a related test in later versions, run in three networks, Sonnet 3.5 agents could carry out single actions, but a Sonnet 3.5 planner using only Sonnet 3.5 agents for every action succeeded in none of the three (v3, Finding 4 and Figure 14; v4, Finding 4 and Figure 15).
- Older models. The models are from 2024. The later "AI models on realistic cyber ranges" post reports how newer Claude models did on these networks without the toolkit.
Later versions of the paper
The arXiv paper changed after the version the post cites. Version 3 (16 May 2025, before the post) is titled "On the Feasibility of Using LLMs to Autonomously Execute Multi-host Network Attacks", names the benchmark MHBench, uses Claude Sonnet 3.7 Thinking, GPT-4o and Gemini 2.5 Pro as baselines, tests nine models with Incalmo, and reports different failure shares (47% to 90% irrelevant commands, 6% to 41% wrongly implemented; v3, Section III-B). In v3 the one partial success without Incalmo is Sonnet 3.7 Thinking taking 11 of 25 files in the 4-layer chain (Section II-C, Figure 2). Version 4 (22 November 2025) is retitled "Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks", reframes the work as red teaming, expands MHBench to 40 networks, and reports that Incalmo with Claude Sonnet 4 reached at least one critical asset in 37 of 40 networks against 3 of 40 for the best shell-based baseline with the same model (v4, Finding 1.A, Figure 2). The headline pattern holds across versions; the models, counts and percentages differ, so a number should be cited with its version. The post, published a month after v3, reports the v1 and v2 results, while Anthropic's Claude Opus 4 and Sonnet 4 system card (May 2025, Section 7.4.6, footnote) already cites the paper by its v3 title.
Research questions
- How much does an active defender change the result? The networks had no monitoring or response. Running the same tasks with detection and blocking in place, and reporting success and time to detection, would show how far the result carries to defended networks.
- How much of the success is the hand-built exploit library? Incalmo's agents choose exploits from a library fitted to these networks. Running the toolkit with a general exploit database, or with model-written exploits for every action, would separate planning ability from the library.
- Is best-of-five the right summary? Reporting the full per-trial distribution and the cost of repeated attempts would show what a single attempt can do and what persistence buys.
- Does the interface effect shrink as models improve? The 2026 follow-up says newer Claude models need the toolkit less. Testing several model generations both with and without Incalmo on the same networks would measure the gap over time.
- Do the results transfer to networks the designers did not build? Testing on networks designed independently of the toolkit's authors (the paper suggests real or proprietary networks) would check whether the 5-of-10 figure depends on the benchmark.
- How reliable is the attack-graph coverage measure? Coverage is scored against one hand-written reference attack per network. Comparing it with an independent labeling of the same logs would estimate how many alternative routes it misses.
How it connects
- Follow-up post. AI models on realistic cyber ranges (January 2026) reports that Claude Sonnet 4.5 succeeded on some of these networks with only a shell, and cites this post as the earlier stage.
- System cards. The Incalmo ranges are part of Anthropic's own cyber evaluations from Claude 4 on: the Claude Opus 4 and Sonnet 4 system card runs Opus 4 on three ranges with the Incalmo harness (Section 7.4.6), and the Claude Sonnet 4.5 system card drops the harness (Section 5.3.2).
- Other cyber evaluations. Building AI for cyber defenders (September 2025) refers to the Equifax simulation as evidence that models had become useful for security work. This note adds Detailed cyber evaluations of Claude 4 (July 2025), which reports a partner's CTF and network challenges for the same period; the post does not link it.
References
- Anthropic Frontier Red Team, with Carnegie Mellon University's CyLab. (2025, June 13). LLMs with cyber toolkits can conduct multistage cyber operations on business-sized computer networks. https://www.anthropic.com/research/cyber-toolkits (first published at red.anthropic.com/2025/cyber-toolkits/; Wayback capture of 15 July 2025 read for comparison, same text). record
- Singer, B., Lucas, K., Adiga, L., Jain, M., Bauer, L., and Sekar, V. (2025). On the Feasibility of Using LLMs to Execute Multistage Network Attacks. arXiv:2501.16466v1 (27 January 2025) and v2 (6 March 2025, same results; version of record for this note). https://arxiv.org/abs/2501.16466v2 record · 2 pages
- Singer, B., et al. (2025). On the Feasibility of Using LLMs to Autonomously Execute Multi-host Network Attacks. arXiv:2501.16466v3 (16 May 2025). record · 2 pages
- Singer, B., et al. (2025). Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks. arXiv:2501.16466v4 (22 November 2025). record · 2 pages