Evaluating Claude's bioinformatics research capabilities with BioMysteryBench

benchmarksbioinformaticsai for scienceagentsevaluationclaude mythos preview

A Science blog post introducing BioMysteryBench, 99 expert-written bioinformatics questions on real datasets whose answers come from verified properties of the data rather than from scientists' conclusions; recent Claude models roughly match a panel of experts on the 76 questions the experts could solve and solve some of the 23 they could not, with Claude Mythos Preview reaching 30% on that set, though many of those wins are not reproducible across attempts.

Anthropic (post by Brianna, a researcher on the discovery team) · 2026-04-29 · original

Benchmarks for scientific ability began with knowledge questions and have moved toward agentic tasks in which a model reads papers, queries databases and analyzes data. This post, published on Anthropic's Science blog on 29 April 2026 and introduced as written by "Brianna, a researcher on the discovery team" (the post gives no surname), argues that biology is particularly hard to benchmark and introduces BioMysteryBench, a bioinformatics benchmark designed around those difficulties. It reports how several Claude models compare with human experts and what strategies Claude used. The post is a blog essay with results shown mostly in figures; this note gives the numbers its text states and describes the rest.

Why biology is hard to benchmark

The post names three problems, each illustrated with the long search for predictors of response to the diabetes drug metformin:

  • Many valid methods. A question can be approached in very different ways, for example a genome-wide association study or gut microbiome sequencing, depending on the scientist's skills and resources. The post credits BixBench with grading conclusions rather than methods, but notes that those conclusions come from one scientist's chain of subjective choices.
  • Subjective decisions in noisy data. Small differences in study design can reverse conclusions. The example is a genetic variant reported in 2011 to predict metformin response, not found in a 2012 study of pre-diabetics, and judged real but smaller in a 2012 meta-analysis. The post credits SciGym with avoiding this by using a simulated lab with a known ground truth, while noting that performance in simulation may not track performance on real data.
  • Questions humans cannot yet answer. The most valuable tasks are ones no human has solved, but a benchmark built from expert answers cannot include them. ProteinGym and CASP use experimental measurements as ground truth for this reason, but cover a narrow set of tasks.

What the benchmark is

BioMysteryBench has 99 questions from several fields of bioinformatics, written by domain experts. Each expert gathered a dataset and wrote a question about "controlled, objective properties of the data" rather than an open scientific conclusion. Answers come from the data's construction or from metadata validated by an independent method; the post's examples are the organism a crystal structure belongs to, and the virus infecting a patient as confirmed by a PCR assay. Most questions use raw or lightly processed DNA or RNA sequencing data (whole-genome sequencing, single-cell RNA-seq, methylation, ChIP-seq, metagenomics, Hi-C), with some from proteomics and metabolomics. Sample questions include identifying which organ a single-cell dataset comes from, which gene was knocked out, and which samples are a given sample's parents.

Each question author also submitted a validation notebook showing that the signal exists in the data, on the principle that checking an answer is easier than finding it. This is meant to exclude unsolvable questions while keeping ones that may be solvable only by AI.

Claude works on each question in a container with a minimal set of standard bioinformatics tools, can install more through pip and conda, and can download resources such as reference genomes from databases including NCBI and Ensembl. Grading checks only the final answer, so any analytical route counts.

The post lists the design's properties as method-agnostic grading, objective ground-truth answers, and the possibility of "superhuman" questions, since answers do not depend on a human being able to solve them.

Human baselines

Up to five domain experts attempted each question from scratch. A question that at least one expert answered correctly counted as human-solvable; there were 76. For the rest, the team considered three explanations: a broken question, a question whose signal is not in the data, or a solvable question beyond the experts' knowledge. After review, 4 broken questions were removed, leaving 23 human-difficult questions.

What it finds

Accuracy (Figures 1 and 2)

Each model attempted each problem five times. On the human-solvable set, accuracy is plotted in Figure 1; the post's summary is that current models "perform on par with human experts" and that the newest generations solve most human-solvable problems reliably. On the human-difficult set (Figure 2), Claude Sonnet 4.6 and more capable models solved "significant fractions", with Claude Mythos Preview "topping out at a 30% solve rate". The post presents the gains as rapid across model generations.

Claude's strategies

Sometimes Claude followed the same approach as the human experts. In other cases it took a different route: where experts used algorithms or databases to annotate a dataset, Claude sometimes recognized patterns or sequences directly. From transcripts of Claude Opus 4.6, the post identifies two main strategies that set Claude apart from the human experts:

  • Knowledge. Opus drew on what it knows about structural biology, molecular profiles and published meta-analyses, solving directly some tasks that would need a human to assemble a meta-analysis or combine databases. The post also shows one case, in the human-solvable set, where prior knowledge led Claude to a wrong answer.
  • Converging methods. When uncertain, Opus 4.6 often tried several approaches and chose the answer they agreed on.

Reliability of solutions (Figure 3)

The post invited Claude Mythos Preview to analyze the results itself and quotes its analysis. Mythos Preview broke each model's solved problems down by how many of the five attempts succeeded. Its reported figures: on human-solvable problems, Opus 4.6 solved 86% of the problems it solved at all in at least four of five attempts, but on the human-difficult set this fell to 44%, and the share of "brittle" wins (one or two of five) rose from 9% to 44%. For Sonnet 4.6 the shift was larger (75% to 22% reliable; 9% to 56% brittle). The analysis describes "the 77.4%→23.5% headline drop" as understating the change, and says Opus 4.7 and Mythos Preview move the frontier slightly (Mythos Preview solves 94% of its human-solvable wins at least four times in five) while the same split holds on the hard set. Figure 3's caption concludes that difficult-set wins "are often lucky reasoning paths rather than reliably reproducible solutions". These numbers are Mythos Preview's, as quoted in the post.

The author judged the analysis sound but "a little…boring?", adding nuance without asking a new question, and describes the models as showing "the seeds of research taste".

The post notes that Genentech and Roche released CompBioBench, 100 computational biology tasks with single ground-truth answers built partly from synthetic data and scrambled metadata, while the post was being finalized. It reports that Claude Opus 4.6 scored 81% overall and 69% on the hardest questions there. These figures are as stated in the post.

Limitations and open questions

  • Unsolved questions stay ambiguous. The post states that for questions neither humans nor models have solved, there is no way to be sure whether they are impossible or only very hard; validation notebooks show the signal exists but not that it can be found from scratch.
  • Reliability. By Mythos Preview's analysis, many wins on the hard set occur in only one or two of five attempts, so single-run accuracy on that set overstates dependable capability.
  • Size. The hard set has 23 questions, so a 30% solve rate rests on a small number of problems.
  • Human baseline. "Human-solvable" means at least one of up to five experts succeeded; the post does not report per-expert accuracy in its text.
  • Prior knowledge. The knowledge strategy that helped on hard tasks also produced at least one wrong answer, and the post does not measure how often.

Research questions

  1. How much does pass rate across five attempts change the ranking of models? Reporting accuracy alongside the share of problems solved at least four times in five, for every model, would show whether reliability and accuracy rank models the same way.
  2. When does prior knowledge mislead? Classifying every wrong answer by whether the transcript shows an answer reached from recalled literature rather than from the data would measure the failure the post illustrates with one case.
  3. Are some human-difficult questions unsolvable in practice? Tracking which of the 23 questions any model ever solves, across future model releases, would separate hard questions from ones whose signal cannot be recovered.
  4. Does converging on agreement between methods improve accuracy? Comparing transcripts where Claude used several methods with those where it used one, on questions of similar difficulty, would test whether the strategy the post highlights pays off.
  5. How well does BioMysteryBench performance predict usefulness to working scientists? Comparing model rankings on the benchmark with ratings from bioinformaticians using the same models on their own data would test the benchmark's validity.

How it connects

References

  • Anthropic (2026). Evaluating Claude's bioinformatics research capabilities with BioMysteryBench. Anthropic Science blog, 29 April 2026. https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench. Dataset preview: https://huggingface.co/datasets/Anthropic/BioMysteryBench-preview