This post asks whether a coding agent can find real bugs in widely used software by writing property-based tests, which state a general rule a function should obey and let a framework search for inputs that break it. Such tests are used less than example tests because someone has to work out the properties; the authors give that job to Claude. The post frames the work as general bug-finding rather than security research, while noting that many logic bugs behind security vulnerabilities are the kind such tests can catch.
The post was published on 14 January 2026 on the Frontier Red Team's blog, red.anthropic.com (first archived copy 15 January 2026, 01:09 UTC), and now appears on anthropic.com with the same date and text, under the shorter title "Finding bugs with Claude and property-based testing." It summarizes a paper, "Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem" (arXiv:2510.09907v1, 10 October 2025), presented at the NeurIPS 2025 Deep Learning for Code workshop and arising from a MATS project. The paper is the version of record for the first evaluation phase (Claude Opus 4.1); the post adds a second phase with Claude Sonnet 4.5 that the paper does not cover. The authors' bug-browsing site (data files last modified 13 January 2026) and the code repository (github.com/mmaaz-git/agentic-pbt, last commit before the post 7ac6a15f89, 3 November 2025) are also sources.
What the post and paper do
Property-based testing. A property-based test states an invariant, such as "the output of a sort is in order" or "decoding an encoded value returns the original," together with a description of valid inputs. The testing framework, here Hypothesis (a Python library whose 2019 paper, cited in the authors' paper, lists Hatfield-Dodds among its authors), generates many inputs, in a way similar to fuzzing, and searches for one that violates the property (post, Introduction).
The agent. The agent is a custom command for Claude Code, Anthropic's terminal coding agent; in the paper's words it "is implemented as a natural language prompt, stored in a Markdown file" (Section 2). Given a target (a file, a module such as numpy, or a single function), it follows six steps: work out what the target is, read its code and documentation, propose properties grounded in what it read, write Hypothesis tests, run them and triage the results, and write a bug report if it is convinced a failure is a real bug. A to-do list tracks progress, and the agent asks of a failing test whether it found a bug or a flaw in its own test, and of a passing test whether it checks anything worthwhile. The post's one instance: a test passed until the agent noticed it had wrapped the whole test in a try-except block.
Phase one (the paper). The authors chose 100 packages: 15 from the Python standard library, 15 popular third-party packages chosen by hand (NumPy, pandas, SciPy, requests and others), and 70 sampled at random from the 5,000 most-downloaded packages on PyPI (Section 3). The agent ran on each package's main module and every submodule one level down, 933 modules in all, using Claude Opus 4.1, in isolated environments with up to 20 agents running at once.
How the bug reports were judged. Claude Opus 4.1 first scored every report against an initial rubric meant to remove clear false alarms. The authors then drew 50 reports at random from the top 80% by score, and two authors judged each independently on two questions: is this a valid bug, and would we report it to the maintainers? They settled disagreements by discussion (Section 3, "Evaluation"). (The post describes the 50 as "manually selected"; the paper's account of a random draw is followed here.) From that review they built a final rubric scoring reports out of 15, had Opus 4.1 score all reports again, and reviewed every report that scored 15 out of 15 by hand.
Phase two (post only). The authors ran the agent with Sonnet 4.5 several times on each of "a subset of 10 important packages," and built a second agent, also Sonnet 4.5, that reads the code and the bug report to judge correctness and severity (post, "Evaluating the agent"). Three paid expert reviewers checked high-severity bugs; the post's introduction adds that they spent "an average of one hour of review per bug," that any bug a reviewer was unsure of was dropped, and that the authors reviewed the rest before filing. The post gives no counts for this phase. The site's Sonnet 4.5 data file holds 1,832 reports across 14 packages, each linked to an analysis file from the evaluation agent and labeled bug (1,165), won't fix (250), documentation fix (223), invalid (169) or feature (25). Eighty reports, all labeled high severity, carry expert votes: three reviewers on 51, two on 15 and five on 14, from 14 reviewer IDs in all (this note's count from the released file; the post's "10 packages" and three reviewers, and the file's 14 packages and two to five reviewers, have not been reconciled).
What it finds
- Volume and cost. The Opus 4.1 agent wrote 984 bug reports, flagging problems in 786 of the 933 modules (84.2%) (paper, Section 4, "Summary statistics"). The runs took 136.6 hours and cost $5,474.20 in API fees, $5.87 per module (range $0.65 to $15.42) and $5.56 per bug report, using 2.21 billion tokens in all.
- Validity of a sample. Of the 50 sampled reports, 56.0% were judged valid bugs (95% CI 42.2% to 69.8%) and 32.0% valid and worth reporting (19.1% to 44.9%) (Section 4, "Manual review"). Both intervals equal the normal-approximation (Wald) interval for 50 reports (this note's arithmetic), the simplest interval for a proportion and one that covers the true rate less often than stated at small samples. Agreement between the two authors before discussion was low: a kappa of 0.31, which the paper reports without naming the statistic further. The paper turns the first figure into a false discovery rate between 30.2% and 57.8% (Section 5, "Limitations").
- Validity after ranking. Of the 21 reports that scored 15 out of 15 on the final rubric, 18 were valid (86%) and 17 were valid and worth reporting (81%) (Section 4). The post calls the ranking step "considerably effective." The 21 are the reports the rubric ranked highest, judged by the same authors who built the rubric, and the paper does not report how many valid bugs ranked below the cut. The released Opus 4.1 file, dated three months after the paper, holds 24 reports scored 15 out of 15 (this note's count); the paper does not say whether the rubric scores were later rerun.
- Cost per valid bug. Dividing $5.56 per report by the 56% validity rate, the paper estimates $9.93 per valid bug, which it calls an upper bound because developers would aim the agent more selectively (Section 5).
- Kinds of bug. The paper reports failures from serialization to numerical precision to faulty caches (Abstract). On the site's Opus 4.1 file, the agent's own labels are 441 logic bugs, 343 contract violations (behavior that departs from documentation or type hints) and 200 crashes (this note's count).
- Self-reflection by model. The post reports "a notable improvement in self-reflection with Opus 4.1 and Sonnet 4.5, compared to Sonnet 4," with no measurement given.
Maintainer validation
The authors reported five bugs by hand, four with patches they wrote, choosing them as "particularly interesting" (paper, Section 4; post, "Maintainer validation"). The post calls the maintainers "the ultimate arbiter of truth." Their outcomes, checked against the repositories:
- NumPy.
numpy.random.waldsometimes returned negative samples, which a Wald distribution cannot produce. The agent's test checked that all samples are positive. The agent's suggested fix was not correct, as the post says; the authors traced the error to catastrophic cancellation (a loss of precision when nearly equal numbers are subtracted) and submitted a more stable formula. The pull request (numpy#29609) was merged on 10 October 2025; the reviewing maintainer reported that the regression test submitted with the patch passed on the unpatched code in the maintainer's build until its mean was raised from 1e8 to 1e9. The post's statement that the new formula has "nearly ten orders of magnitude lower relative error" rests on a plot a maintainer posted in the pull request, which this note has not checked. - aws-lambda-powertools. A function that splits a dictionary into chunks returned the first chunk repeatedly; the agent tested that the chunks recombine into the original. Patch merged on 28 August 2025 (#7246).
- cloudformation-cli. A hashing function gave every list the same hash, because Python's in-place sort returns nothing; the agent tested that different inputs hash differently. The post lists the patch as "submitted" (#1106). The pull request was closed without being merged on 7 May 2026 by its author, with no comment in the thread.
- tokenizers. A color function returned CSS color strings missing a closing parenthesis; the agent tested the output against the format. Patch merged on 29 August 2025 (#1853).
- python-dateutil.
easter()returned non-Sunday dates under the Julian calendar method. The maintainer explained that the method returns the Julian-calendar date in a type that assumes the Gregorian calendar, and closed the issue as working as intended (#1437, 28 August 2025).
The authors wrote the patches themselves; the cloudformation-cli pull request states that the property "was proposed and written 'autonomously' by the agent" and that the author "wrote this PR myself."
The written instructions
The agent is a written prompt, printed in full in the paper's Appendix A and released in the repository. The paper says it was developed "by using industry best-practices," material from the Hypothesis documentation, and "iterative refinement from manual inspection of agent runs" (Section 2). Its instructions shape what counts as a bug:
- Properties must come from the code's own claims: "Only test properties that the code is explicitly claiming to have," in the docstring, comments or how other code uses it, and "Proposed properties should be strongly supported by evidence" (Appendix A, line 86).
- The prompt lists kinds of property to look for (invariants, round trips, inverse operations, metamorphic relations and others) and tells the agent to study a function's callers to learn its valid inputs.
- A triage rubric asks whether a failure reproduces, whether the input is realistic, whether the property is actually claimed and whether real users would be affected.
- A closing reminder reads: "Quality over quantity. One real bug discovery > 100 passing tests." (Appendix A, line 288).
The paper says the authors "attempt to reduce false alarms" through the emphasis on evidence, careful reflection and a few high-value properties (Section 2, "Key Design Choices"), and the post states that the self-reflection loop and the grounding of properties in documentation help reduce them, but neither reports a comparison of prompt versions, so the effect of any one instruction is not measured. The scoring rubrics used to rank reports are in the repository, not in the paper.
Limitations
- Most reports were not reviewed. The validity rates come from 50 sampled reports and the 21 top-ranked ones, judged by two authors who agreed at a kappa of 0.31 before discussion; the other reports were not reviewed by hand (paper, Section 5).
- Intent ambiguity. The agent cannot tell an intended design choice from a bug. The paper's example is a
requestsclass that looks like a dictionary but was never meant to behave like one; the dateutil report is another, and the post concludes that "deriving properties from code with subtle or complex semantics remains difficult." - Selected cases. The five reported bugs were chosen as interesting, and the second phase reports no counts, so neither shows a rate of bugs that maintainers accept.
- Graders inside the pipeline. Opus 4.1 both wrote the reports and scored them for the ranking, and in phase two a Sonnet 4.5 agent checked a Sonnet 4.5 agent's reports; the human reviews are the check on both.
- Comparison with earlier work. The paper contrasts its results with an earlier study in which at most 41% of generated tests ran and passed and at most 21% of documented properties were captured (Vikram et al., 2023), but that study used different functions and measures, so the contrast is not a head-to-head comparison.
- Peer review. The paper is a four-page workshop paper, with arXiv v1 the only version.
Research questions
- How many of the agent's reports would maintainers accept? Filing a random sample of reports, rather than selected ones, and recording acceptance, rejection and no response would measure precision against the authors' own "arbiter of truth."
- Which prompt instructions reduce false alarms? Running the agent with and without the evidence requirement and the triage rubric on the same modules, and scoring the reports blind, would show whether the written instructions reduce false alarms, as the paper intends and the post states.
- Does the 15-out-of-15 cut discard many valid bugs? Hand-reviewing a sample of reports scored below the cut would estimate the recall of the ranking step.
- How much does model choice matter? Running Opus 4.1, Sonnet 4.5 and Sonnet 4 on the same modules, with the same review, would test the post's statement that self-reflection improved.
- What share of found bugs bear on security? Classifying validated bugs by whether they could be exploited would test the post's suggestion that the method can find vulnerabilities before deployment.
- Would independent reviewers agree more with a written definition of a bug? Re-coding the 50 sampled reports with a fixed codebook by reviewers outside the project would show whether the kappa of 0.31 reflects the reports or the criteria.
How it connects
- Smart contracts. The post cites AI agents find $4.6M in blockchain smart contract exploits (December 2025) as earlier work on smart-contract bugs, noting that almost all of those vulnerabilities were logic bugs.
- Defensive use of Claude. Building AI for cyber defenders (September 2025) describes training Claude Sonnet 4.5 for finding and patching vulnerabilities, the defensive aim this post's conclusion shares; this connection is the note's.
- The models. The agent ran on Claude Opus 4.1 (Claude Opus 4.1 system card addendum, cited by the paper) and Claude Sonnet 4.5 (Claude Sonnet 4.5 system card).
- Statistics. The intervals on the 56% and 32% rates are Wald intervals for a proportion from 50 reports (this note's identification); Confidence Intervals for a Proportion compares that interval with the Wilson and exact intervals.
References
- Maaz, M., DeVoe, L., Hatfield-Dodds, Z., & Carlini, N. (2026, January 14). Finding bugs across the Python ecosystem with Claude and property-based testing. Anthropic Frontier Red Team. https://www.anthropic.com/research/property-based-testing. Originally at https://red.anthropic.com/2026/property-based-testing/. record
- Maaz, M., DeVoe, L., Hatfield-Dodds, Z., & Carlini, N. (2025). Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem. NeurIPS 2025 Workshop on Deep Learning for Code. arXiv:2510.09907v1. https://arxiv.org/abs/2510.09907 record
- Agentic property-based testing code. https://github.com/mmaaz-git/agentic-pbt, commit
7ac6a15f89(3 November 2025). record - Bug report browser and data files (
data-opus-4.1.json, 984 reports;data-sonnet-4.5.json, 1,832 reports). https://mmaaz-git.github.io/agentic-pbt-site/, files dated 13 January 2026. record - Pull requests and issue: numpy/numpy#29609; aws-powertools/powertools-lambda-python#7246; aws-cloudformation/cloudformation-cli#1106; huggingface/tokenizers#1853; dateutil/dateutil#1437 (GitHub, read November 2026).
- Vikram, V., Lemieux, C., Sunshine, J., & Padhye, R. (2023). Can large language models write good property-based tests? arXiv:2307.04346. Cited in the paper; not read for this note. record