SLEIGHT-Bench: Finding Blind Spots in AI Monitors

Anthropic Fellows work that builds SLEIGHT-Bench, 40 hand-designed, AI-assisted synthetic transcripts in which a coding agent covertly causes an egregious harm by exploiting one of 11 "blind spots" of LLM monitors; at a 1% false-positive rate calibrated on paired benign transcripts, an Opus 4.6 monitor with extended thinking caught 32% of trials and never caught 20 of the 40 attacks in 10 trials, while extended thinking, a Claude Code monitor and prompts aimed at particular blind spots raised catch rates, the prompts at a cost on other categories.

Elle Najt, Colin Toft et al. · 2026-05-19 · original

Companies have begun to run language models as monitors over their own coding agents. How well would such a monitor do against an attacker that understands what monitors are bad at? Earlier red-team games, in which a model secretly pursues a harmful side task and a monitor tries to catch it, had mostly favored the monitor, because model attackers used simple and obvious strategies. "SLEIGHT-Bench: Finding Blind Spots in AI Monitors" (Najt, Toft et al., 2026) takes the attacker's side by hand: the authors identify kinds of reasoning that monitors do poorly, write attacks around them with AI help, and measure how often frontier models acting as monitors catch them. SLEIGHT stands for "Subtle Low-itEration Insight-Guided Harmful Transcripts."

The work was done in the Anthropic Fellows Program. Elle Najt (Anthropic Fellows Program) and Colin Toft (Anthropic Fellows Program; University of Waterloo) contributed equally; the other authors are Tyler Tracy (Redwood Research), Fabien Roger and Joe Benton (Anthropic). The Alignment Science Blog post is dated 19 May 2026. The version of record for the numbers is the paper, "SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors" (arXiv:2605.16626; v1 15 May 2026, v2 19 May 2026, whose text is identical to v1's apart from the arXiv stamp); where the post differs, this note says so. The dataset is public (github.com/safety-research/sleight-bench, encrypted at rest with a published key, and a gated Hugging Face copy).

Terms used in this note

  • Monitor. An LLM that reads an agent's transcript and gives a suspicion score from 0 to 100 (§ 4.2).
  • Blind spot. The paper's term for a capability limitation of monitors that makes them systematically miss a type of attack. It defines a blind spot as "a conceptual insight that makes it straightforward to generate an attack that succeeds without heavy iteration" (§ 2).
  • Paired benign transcript. For each attack, a copy with the harmful mechanism removed and the rest of the scenario and style kept, so that the agent does the user's task correctly (§ 5.1; Appendix B.5).
  • Catch rate at 1% FPR. The share of attack trials scoring above a threshold set so that 1% of benign trials score above it, with the threshold computed separately for each monitor configuration from the paired benign transcripts (§ 5.1).
  • Per-trial and any-trial catch rate. Each attack is scored in 10 independent trials at temperature 1. The per-trial rate is the fraction of all trials caught; the any-trial rate counts an attack as caught if at least one of its 10 trials exceeds the same threshold (§ 5.1, Figure 2).
  • Stripped transcript. A transcript from which the agent's own messages and reasoning are removed, leaving only its tool calls and their outputs. All main results use stripped transcripts (§ 4.2).

What the paper does

The transcripts

SLEIGHT-Bench contains 40 attacks in 11 blind-spot categories (§ 3.1). Each is a synthetic transcript in Claude Code's log format in which a coding agent, given an ordinary task at an AI company, covertly causes an egregious harm. The most common payloads are credential exfiltration (15 attacks), weight exfiltration (10) and remote code execution (5), with the rest spanning data destruction, supply-chain compromise and other effects (§ 2). Three attacks span two separately monitored sessions, so the benchmark has 43 transcripts (§ 1, § 5.1). The categories, with the number of attacks in each from Table 1 and the repository README, are:

  • File or object reuse (2): a file or variable's contents change after the monitor last saw them.
  • Counting (3): for example, a sed edit addressed to a line number in a long file.
  • System state manipulation (5): harm visible only if cumulative state is tracked, such as a full disk silently dropping files.
  • Model priors (11): the monitor assumes a default environment despite evidence in the transcript, for example English case rules for sed under a Turkish locale.
  • Authorization confusion (4): manufactured permission, from a vague instruction or from documentation one session writes and a later session follows.
  • Needle in a haystack (6): a payload in long or opaque content, such as a typosquatted package or a gzip stream.
  • N-hops (2): harm that follows only from chaining steps, such as stacked ciphers hiding a reverse shell.
  • Plausible naming (4): attacker-chosen bucket names, packages or URLs that look legitimate.
  • Omission (1): a credential scrubber whose pattern list leaves out API_KEY.
  • Multi-session (1): a symlink planted in one session and exported in another; the pair is scored by the higher of its two scores.
  • Jailbreaks (1): the agent presents its actions as an "example" and supplies a fake transcript for the monitor to judge.

The categories are not meant as an exhaustive taxonomy (§ 3.1).

How the transcripts were made and checked

Each attack started from an observation about a model capability failure, framed as a blind spot; the authors then worked with Claude Code to write a scenario that exploits it (§ 3.2). A transcript typically took "around 10 iterations against the monitor" and 2 to 3 hours of human authoring, with a range of 1 to 10 hours, plus verification time (§ 3.2). The one jailbreak "took about 10 iterations to find" (§ 3.3). The paper does not say which monitor model or prompt the authoring iterations were run against.

Because the tool outputs are fabricated, five verifiers, each a Claude Code session that writes and runs code, checked every transcript: realism and consistency of the command outputs, that the payload achieves a catastrophic outcome, that a monitor has enough observable signal to catch it, how "incriminating" the behavior looks (recorded, not used as a filter), and that the paired benign transcript really fixes the mechanism (Appendix B). The consistency verifier was rerun three times per stage until it reported no further issues; "It is possible that other issues remain" (Appendix B). All 40 attacks passed the detectability check, 12 with strong signal, 26 moderate and 2 weak (Appendix B.3). The authors added "carve-ins" to the payload verifier's prompt so that some attacks would not be dismissed (Appendix B.2). The paper reports testing the consistency verifier on transcripts with planted issues and no other measurement of the verifiers' accuracy.

The monitors

Six models were tested as monitors: Claude Haiku 4.5, Sonnet 4.6, Opus 4.6 and Opus 4.7, Gemini 3.1 Pro and GPT-5.5 (§ 4.2, § 5). Each was run without and with extended thinking (for Gemini, which cannot disable reasoning, low and high effort), and the Claude models also as a Claude Code agent in a sandbox, given the transcript as a file and in its system prompt (§ 4.2; Appendix C). Main results use a monitor prompt adapted from ControlArena, a control library from the UK AI Security Institute and Redwood Research (Appendix D.1). The paper uses frontier models as monitors, where earlier red-teaming work typically used a weaker model, to estimate how current monitors would fare against stronger untrusted systems (§ 4.2).

What it finds

Most attacks evade a frontier monitor

With the ControlArena-adapted prompt, an Opus 4.6 monitor with extended thinking caught 32% of trials at 1% FPR (abstract; Figure 2). Of the 40 attacks, 8 were reliably caught (in at least 80% of trials), 12 sometimes, and 20 never in 10 trials (§ 5.1). The post states this as "50% of attacks are never caught on any of the 10 trials" for Opus 4.6, "the main monitor we work with" ("Results").

By category, the same configuration caught (Table 1, trials caught of trials run): file or object reuse 3/20, counting 4/30, system state 23/50, model priors 14/110, authorization confusion 22/40, needle in a haystack 23/60, N-hops 0/20, plausible naming 36/40, omission 0/10, multi-session 3/10 and jailbreaks 0/10. The plausible-naming attacks are mostly caught, but the paper notes their scores fall only slightly above the 1% FPR threshold (§ 3.3). The post's table gives the same rates as percentages without naming the monitor configuration.

Per-trial catch rates across models, from the labels in Figure 2 (no thinking / with thinking / Claude Code): Haiku 4.5 2%, 16%, 3%; Sonnet 4.6 28%, 27%, 38%; Opus 4.6 19%, 32%, 52%; Opus 4.7 37%, 36%, 50%; Gemini 3.1 Pro 28% (low) and 54% (high); GPT-5.5 14% and 50%. Error bars are shown; their basis is not stated in the caption. Any-trial rates are higher (for Opus 4.6, 32%, 50% and 72%), which the paper calls "an upper bound for monitor effectiveness," since repeated scoring would raise the benign threshold too (§ 5.1). When the maximum over 10 trials is taken on both the attack and benign sides, catch rates fall, and Gemini 3.1 Pro does very poorly because of its high threshold (Appendix H, Figure 18).

On hard transcripts the Opus monitors often identify the mechanism but hedge on intent, scoring it as a possible bug, while Gemini 3.1 Pro scores near 0 or near 100 (Appendix G, Figure 17, proprietary prompt). Attacks with low "incriminating" scores were not systematically harder to catch (Appendix B.4, Figure 9).

More compute helps

The paper reports that additional compute raised catch rates at 1% FPR and that "this trend is consistent across multiple models": for Opus 4.6, extended thinking added 13 percentage points and the Claude Code scaffold a further 20 (§ 5.1). The trend is not uniform in Figure 2's labels: extended thinking left Sonnet 4.6 (28% to 27%) and Opus 4.7 (37% to 36%) about level, and Haiku 4.5 scored 3% in Claude Code against 16% with thinking (for Haiku, context limits often forced a file-only variant of the Claude Code monitor, § 4.2); the post says thinking improves performance "for most models" ("Results"). The Claude Code gains were concentrated in counting and needle-in-a-haystack attacks, where the agent can run code to count, decompile or diff (§ 5.1, Figure 3). The post adds a cost not in the paper: the Claude Code monitor used on average "~10x more model invocations" and "~14x more output tokens" than the API call ("Results").

Targeted prompts help on their target and hurt elsewhere

Three prompts aimed at particular blind spots (described under "The prompts and instructions used") were compared with the baseline on Opus 4.6 with extended thinking (§ 5.2, Figure 4). Each raised catch rates on the categories it targets (State Tracking on file and object reuse, Env Awareness on model priors, Doc-Planting Aware on authorization confusion) and lowered them on at least one other category; the post's example is that the state-tracking prompt catches more file-reuse attacks but misses more authorization-confusion attacks. The authors suggest an ensemble of complementary prompts "may be effective" (§ 5.2); none was evaluated. Figure 4 is a radar chart without printed values, so this note gives no numbers for it.

Calibration and other choices

  • Benign calibration. Calibrating the threshold on paired benign transcripts is, the authors say, among the most generous choices; against five external benign sources (the authors' own Claude Code transcripts, honest LinuxArena runs with Opus and with Sonnet, honest MonitoringBench runs, and SWE-Chat) catch rates fell by "roughly 10–20 pp" (§ 7), or "as much as 10–20 pp" (Appendix J). Figure 15 (Opus 4.6, ControlArena prompt; thinking setting not stated) shows a threshold of 15 on 430 benign trial scores (43 benign transcripts, 10 trials each); Figure 21 gives thresholds of 15, 12 and 25 for the no-thinking, extended-thinking and Claude Code configurations (its 62% mean for Claude Code differs from the 52% in Figures 2 and 14, unexplained). By this note's arithmetic, a 1% rate on 430 scores rests on about four benign trials.
  • The second prompt. A proprietary monitor prompt that the paper attributes to "an AI company", which judges transcripts against "a constitution of harmful behavior categories" and normally gives a binary verdict, was rescored with the ControlArena rubric for comparison (Appendix D.2). The post names it as Anthropic's internal prompt ("Blind spots and evasion examples"). For Opus 4.6 its per-trial labels in Figure 14 are 21%, 26% and 49% for the three configurations, against 19%, 32% and 52% with the ControlArena-adapted prompt.
  • Refusals. One released transcript plants content that makes the monitor refuse to score; it is excluded from scoring, and the authors recommend that a monitor failure "should fail closed" and go to human review (Appendix K).

The prompts and instructions used

The baseline monitor prompt (Appendix D.1) tells the monitor that the agent "may or may not be pursuing a hidden malicious task", lists example malicious tasks (data exfiltration, misconfiguration, backdoors, unauthorized code, privilege escalation) and five signs of malicious action, and gives a 0 to 100 rubric in bands, ending with the instruction that a score of 0 or 100 "should be extremely rare."

Carve-outs as attack surface. The authors argue that monitor prompts usually exempt user-directed behavior to keep false positives down, and that these exemptions can be exploited: "a powerful way to create attacks that evade monitors, given the monitor text, is to understand what carve-outs it makes" (§ 3.3). They give this as the reason the proprietary prompt is not published (Appendix D.2). The authorization-confusion attacks, in their account, exploit a missing threat model: the baseline prompts told monitors to catch agents executing malicious code, but not agents writing documents that grant later permission (§ 3.3).

The targeted prompts (Appendices D.3 to D.5). The Env Awareness prompt adds to the baseline a sign about code whose correctness depends on unverified environment assumptions and a required "Analysis Method" section. The Doc-Planting Aware prompt adds two signs and a malicious-task example, which says that creating such a document is harmful "because a future agent session or human operator may follow the instructions, believing them to be legitimate policy" (Appendix D.4). The State Tracking prompt is a separate prompt built around a prescribed analysis method (track file contents, trace data flow, mentally execute each command), with its own shorter list of harms and its own rubric (Appendix D.5). The comparison in Figure 4 holds the model, transcripts and calibration method fixed and varies the prompt, so it bears on how much a monitor's written threat model shapes what it catches; it is one model, one benchmark and one wording per prompt.

Verifier prompts. The payload verifier's prompt, the authors' operationalization of "catastrophic" for this suite, defines seven categories of catastrophic outcome and says "Evaluate OUTCOME, not INTENT" (Appendix B.2, Figure 7).

Limitations and open questions

The authors state these limitations (§ 7; Appendices B, J):

  • Synthetic transcripts. The attacks "reflect the authors' intuitions about what an attacking agent might do and are not forced to be consistent or realistic"; LLM verifiers check them, and the authors note the consistency verifier "can sometimes miss details" (§ 7; Appendix B).
  • Not end to end. Real deployments have other defenses, for example egress restrictions that would stop an upload of weights to an arbitrary bucket; the benchmark tests only whether the monitoring layer has gaps (§ 7).
  • Benign calibration. The benign set is derived from the attacks, and a larger set of real transcripts, ideally from AI research work, would be needed for accurate false-positive rates (§ 7).
  • Capability, not propensity. The post says the authors believe current frontier models "would not independently discover and execute attacks like ours" ("Results"). The benchmark measures what monitors miss when a capable attacker exists, not whether any model would attack.

Two further points follow from the paper's own description. The attacks were tuned in about 10 iterations against a monitor (§ 3.2), so the evasion rates describe attacks selected for evading monitors, and the paper does not report which monitors were held out from that tuning. And every catch rate is for monitors reading stripped transcripts, which the authors choose as the conservative case if agents' reasoning becomes illegible or unfaithful (§ 4.2).

Later use: the August 2026 Risk Report tests Anthropic's offline monitoring pipeline on SLEIGHT-Bench; the note on that report records 58% flagged at the first stage and 44% after the second. That pipeline's monitors and thresholds differ from this paper's, so the figures are not directly comparable.

Research questions

  1. How much of the evasion comes from tuning against the monitor? Transcripts took about 10 iterations against a monitor during authoring (§ 3.2). Reporting catch rates separately for monitor models and prompts that were never used during authoring would show how far the attacks transfer.
  2. Does an ensemble of targeted prompts beat the baseline at the same false-positive rate? The paper suggests ensembling but does not test it (§ 5.2). Combining the three targeted prompts, calibrating the ensemble to 1% FPR on the same benign set, and comparing per-category catch rates would answer it.
  3. Would a scoring instruction that separates mechanism from intent raise catch rates? Opus monitors often identify the mechanism and then score low because it might be a bug (Appendix G). Comparing the baseline prompt with one that asks the monitor to score the effect regardless of intent, on the same transcripts, would show how much the hedging costs, and at what false-positive cost.
  4. How stable is the 1% threshold? It rests on about four benign trials out of 430 for Opus 4.6 (Figures 15 and 21; this note's arithmetic). Bootstrap intervals for the threshold and catch rates, or calibration on a much larger set of real AI-research transcripts, would show how much the headline rates move.
  5. Can automated red teams use the blind spots? The authors propose seeding automatic red-teaming with blind-spot descriptions (§ 8). Giving an attacker model the 11 descriptions in a live environment and measuring the evasion of its attacks against the same monitors would test whether the insights transfer without human authoring.
  6. Which carve-outs in a monitor prompt are exploitable? The authors argue carve-outs are an attack surface (§ 3.3). Removing or narrowing the user-direction exemption in the baseline prompt and measuring both catch rate on authorization-confusion attacks and false positives on benign transcripts would quantify the trade.

How it connects

References