SWE-bench measures whether an AI system can resolve real GitHub issues from popular open-source Python repositories. The system is given the repository as it stood just before the issue was fixed, must change the code, and is graded by running the unit tests from the pull request that originally closed the issue. Because the result depends on the software around the model as much as on the model, a SWE-bench score measures an agent: in the post's definition, a model plus the "scaffolding" that builds its prompts, parses its outputs into actions, and feeds results back in a loop. This post, written by Erik Schluntz, explains the agent Anthropic built for the upgraded Claude 3.5 Sonnet, which scored 49% on SWE-bench Verified, and is meant to help developers get similar performance (introduction). Anthropic's research listing dates the post 30 October 2024, and this note uses that date; the page itself, now in Anthropic's Engineering section, shows a published date of 6 January 2025.
Terms used in this note
- SWE-bench Verified. A 500-problem subset of SWE-bench that humans reviewed to make sure each problem is solvable from the issue text, since some original tasks need context outside the issue (introduction).
- Scaffold. The code around a model that turns it into an agent: prompt construction, tool execution, and the interaction loop.
- Tool description. The natural-language text, together with a schema of parameters, that tells a model what a tool does and how to call it.
- Hidden tests. The grading tests, which the agent cannot see while it works.
The agent
The design goal was to "give as much control as possible to the language model itself, and keep the scaffolding minimal" (section "Tool Using Agent"). The agent has a prompt and two tools, and keeps sampling until the model decides it is finished or reaches its 200k-token context limit. The model chooses its own path through the task rather than following a fixed workflow. For running the benchmark, the agent code was built on the SWE-Agent framework (section "Examples of agent behavior").
The prompts and tool descriptions
The post prints the full prompt and most of the tool specifications, and is direct about the role their wording played.
The task prompt. The prompt tells the model where the repository is, gives the pull request description, says the tests have already been handled and should not be modified, and asks for "minimal changes to non-tests files". It then suggests five steps: explore the repository, write a script that reproduces the error, edit the source, rerun the script, and consider edge cases. It ends: "Your thinking should be thorough and so it's fine if it's very long." The post describes this prompt as a suggested approach that is "not overly long or too detailed", and advises developers who are not sensitive to token costs that explicitly encouraging a long response can help.
The bash tool. Its schema takes only the command, but the description "carries more weight": it tells the model that input need not be XML-escaped, that there is no internet access but there is a package mirror, that state persists between calls, how to view a range of lines, to avoid commands with very large output, and to run long-lived commands in the background.
The edit tool. A tool called str_replace_editor can view, create, replace text in, insert into, and undo edits to files. Its description explains, for example, that the text to replace must match exactly one place in the file, including whitespace, or the replacement is refused.
Revising tool text to prevent errors. The post says the team "put a lot of effort into the descriptions and specs" of these tools across many agentic tasks, tested them for ways the model might misunderstand them, and "edited the descriptions to preempt these problems". It argues that "much more attention should go into designing tool interfaces for models", comparable to the attention given to interfaces for humans. The post gives two examples of this "error-proofing": requiring absolute file paths, which it reports improved performance after the model made mistakes with relative paths once it had moved out of the root directory, and using exact string replacement for edits, which was the most reliable of several editing formats tried, with an error message returned when the target text matches zero or several places.
What it finds
Scores. With this scaffold, the upgraded Claude 3.5 Sonnet scored 49% on SWE-bench Verified, against 45% for the previous state of the art, 33% for the original Claude 3.5 Sonnet and 22% for Claude 3 Opus (section "Results"). The post notes that no model had yet passed 50% on the benchmark at the time of writing.
A worked example. In a scikit-learn issue, RidgeClassifierCV rejected a store_cv_values argument. The model viewed the repository, wrote a reproduction script, confirmed the error, found that the class did not pass the parameter through to its base class, edited the constructor, and submitted after 12 steps; the tests passed (section "Examples of agent behavior").
Behavior. Some tasks took more than 100 turns, and on others the model kept trying until it ran out of context. Compared with older models, the upgraded model self-corrected more often and tried several different solutions instead of repeating the same mistake.
Challenges in running the benchmark
The post lists four (section "Challenges"):
- Duration and cost. Many successful runs took hundreds of turns and more than 100k tokens.
- Grading. Some apparent failures were environment problems, such as installation patches applied twice, rather than model errors.
- Hidden tests. The model often believed it had succeeded when it had not. Some failures solved the problem at the wrong level of abstraction ("applying a bandaid instead of a deeper refactor"), and others solved the problem but did not match the original tests, which the post calls "a little less fair".
- No image viewing. The agent could not look at image files or URLs, which made some tasks, especially Matplotlib ones, hard and prone to hallucination.
Limitations and open questions
The post reports one score per model with no repeated runs or uncertainty, compares with a previous best from another developer's system without describing it, and does not ablate the prompt or tool descriptions, so the contribution of each wording change is described qualitatively.
Research questions
- How much do the tool descriptions contribute? The post credits careful tool text but gives no ablation. Running the agent with minimal descriptions, then adding each instruction back, would measure the effect of each.
- Does encouraging long thinking change the score? The prompt asks for thorough, long reasoning. Comparing scores and token costs with and without that sentence would test the post's advice.
- How often are failures unfair? The post separates wrong-abstraction failures from correct solutions that fail the original tests. Manually classifying a sample of failures would give the share of each.
- How large is the uncertainty on 49% versus 45%? With 500 problems, a standard error and a paired comparison against the previous system would show whether the difference is statistically reliable.
- What does image access add? The post flags missing vision as low-hanging fruit. Adding a file-viewing tool and rerunning the Matplotlib tasks would measure the gain.
How it connects
- Same model. The model card addendum for the upgraded Claude 3.5 Sonnet reports the same benchmark result, 49.0%, with 33.4% for the original Claude 3.5 Sonnet; neither document cites the other.
- Later advice on agents. Building effective agents (December 2024) links this post three times, including for the observation that the team spent more time optimizing tools than the overall prompt and for the switch to absolute file paths.
References
- Schluntz, E. (2024). Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet. Anthropic. Listed 30 October 2024; page dated 6 January 2025. https://www.anthropic.com/research/swe-bench-sonnet
- Anthropic. (2024). Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet, Section 2.2, Table 2. https://www-cdn.anthropic.com/c7822cdc35ad788ec87e14b3a9d45010f1f86c38.pdf
- Anthropic. (2024, December 19). Building effective agents. https://www.anthropic.com/research/building-effective-agents