Paving the way for agents in biology

An unreviewed summary of a Science blog essay by Laura Luebbert, which reports that research agents retrieving viral sequence records from a public database were neither accurate nor reproducible enough for scientific use, and that a deterministic retrieval layer raised their accuracy and removed most of the variation between models.

Laura Luebbert · 2026-06-08 · original

Not reviewed. This note reports what the post itself says. Unlike the other notes in this collection, its claims have not been checked against the underlying papers, reports or data. The work concerns biological design that could be misused, and in-depth analysis of it triggers the safety safeguards of the AI system used to prepare these notes, so it was left at the level of the post's own summary.

What the post reports

The post argues that biological data infrastructure was built for humans clicking through browsers, and that this, rather than reasoning ability, is now the limiting factor for scientific agents. Its analogy is an old city laid out before cars: beautiful, but hard for a modern vehicle to drive through. Software, by contrast, grew up with version control, documented APIs and package managers, which the post offers as one reason coding agents have advanced faster than biological ones. It takes a talk by Andrej Karpathy, in which he said of a web app he had built that "The code was the easiest part! Most of the work was in the browser, clicking things", as the same complaint arriving in software.

The case study is retrieval of viral sequence records from NCBI Virus, a portal over several internationally synchronized sequence databases. The post says much of the portal's filtering logic exists only in its web interface, so a request a virologist can satisfy in a few clicks may take a long script against several APIs. Its benchmark, VirBench, contains 120 realistic queries across 40 pathogens with manually verified ground-truth counts. Left to work with the infrastructure as it stands, the agents tested (the post names Claude Sonnet 4, Claude Opus 4.7, Biomni OSS, Edison Analysis, GPT-5.2-pro and GPT-5.5) reached mean accuracies of 16.9% to 91.3%, where the post argues the practical bar is effectively 100%. Repeated runs of the same query often disagreed with each other. The post illustrates downstream cost with a phylogenetic analysis in which incomplete retrievals shifted the inferred root date of an outbreak, and with an antibody-epitope comparison that gave three different impressions across three runs.

The remedy the post describes is gget virus, a deterministic retrieval layer developed with researchers at NCBI that reproduces the web interface's filtering behavior and logs how each result was produced. With it, the post says, accuracy rose above 90% for every agent, peaking at 99.7% for GPT-5.5, run-to-run variability was largely eliminated, and the gap between models narrowed, so that reliable dataset construction no longer depended on using the newest or most expensive model.

Who did the work and who checked it

The essay is written by Laura Luebbert and is based on a preprint by Ferdous Nasri, Sarah Gurev, Patrick Varilly, Krithik Ramesh, Nuala A. O'Leary, Jonah Cool, Bernhard Y. Renard, Pardis Sabeti and Luebbert; the post links it on arXiv and does not claim peer review. It credits Sarah Gurev and Gage Moreno with the example virology analyses, says gget virus was built in collaboration with researchers at NCBI, and lists a long acknowledgements section of readers and editors, Karpathy among them. The post is published by Anthropic, whose models are among those evaluated.

Caveats the post itself gives

Two of the comparisons are hedged in footnotes: Edison Analysis used older fallback models that could complete the benchmark without triggering biosecurity-related access restrictions, so the post says its results are not directly comparable, and Sonnet 4 is described as the newest Anthropic model available for the evaluation for the same reason. The post states that its example analyses are for illustration only and are not medical or public-health guidance. It also grants that tools like gget virus may approach zero benefit if models improve enough to navigate messy portals unaided, while arguing that even then, cost, speed and auditability would favor a deterministic layer.

How it connects

References