Evals by capability

Practical single-cell RNA-seq analysis tasks — directly in the dbverse / spatial-omics wheelhouse and a high-value target for a bio-specialist leaderboard. The live verified leaderboard now has GPT-5.6 Sol at 62.1% with Pi and Opus 5 at 60.1% with Claude Code.

Real-world spatial biology analysis tasks across five technologies and seven task categories, with a 115-evaluation expert-verified subset. The live verified leaderboard has GPT-5.6 Sol at 74.5% with Codex and Opus 5 at 71.0% with Claude Code.

24 long-horizon spatial biology evaluations spanning CosMx, Visium, Xenium, MERFISH, Slide-seq, histology, and lineage data. The live leaderboard now has GPT-5.6 Sol at 38.9% with Codex and Opus 5 at 25.0% with Pi.

21 long-horizon single-cell evaluations across five research areas, deterministically graded with trajectory rubrics. The live leaderboard now has Opus 5 at 41.3% with Pi and GPT-5.6 Sol at 38.1% with Pi.

Open-answer agentic bioinformatics: agents must run multi-step analyses over real datasets. Scores are low and the open-answer vs multiple-choice gap is large, so treat MCQ numbers with skepticism. Best public agents still miss roughly half the questions.

End-to-end bioinformatics pipelines (RNA-seq, variant calling, metagenomics) with a robustness twist: stress-tests agents under corrupted inputs and decoy files, LLM-graded for pipeline progress and outcome validity. Finds frontier agents complete pipelines without heavy scaffolding but break under perturbation. ICML 2026.

OpenAI's June 2026 benchmark for scientific 'research taste': 129 synthetic genomics / quantitative-biology / translational-medicine problems, each pairing a noisy dataset with a target estimand, graded deterministically from a known causal structure. Extremely hard and unsaturated.

OpenAI's rubric-graded evaluation of medical/health capabilities across realistic healthcare conversations. Model-graded against physician-written rubrics, so scoring cost is nontrivial. Already in inspect_evals.

Clinician-authored visual QA over radiology images — a multimodal medical benchmark. Tests vision-language models, not text-only. Already in inspect_evals.

Molecular-biology reasoning with a hard split; reports accuracy on the human-solved subset. Newer and unsaturated — a useful difficulty signal for frontier bio reasoning.

Broad chemistry knowledge and reasoning over 2,786 QA pairs. Chemistry-adjacent to biology (drug discovery, molecular properties); included as a boundary domain. Already in inspect_evals.

Broad biology-research QA (literature QA, protocols, figure reading, sequence manipulation). Already implemented in inspect_evals, so it is the natural first target to reproduce ourselves with inspect_ai. Verified per-model scores pending our own runs.

FutureHouse's successor to LAB-Bench: ~1,900 tasks in more realistic research contexts, and markedly harder — model accuracy drops 26–46% across subtasks vs the original. Supersedes LAB-Bench (v1) as the current version.

Preliminary 15-task preview of Phylo's trace-based evaluation for long-horizon biological data analysis. It scores analytical process—including data handling, method selection, statistical rigor, source reliability, and reasoning—rather than only final-answer accuracy. Results are preliminary and the benchmark remains a work in progress.

OpenAI's 750 expert-authored tasks spanning 7 scientific workflows × 7 life-science domains, each with an expert rubric — built to capture the ambiguity and judgment calls that knowledge-QA benchmarks miss (complex artifacts, situational ambiguity, open-ended answers). Far from saturated: no model passes 22.8% of tasks, and 34.8% have a best-model pass rate under 20%. Score shown is the problem-weighted normalized rubric score; top pass rate is 36.1% (GPT-Rosalind).

Phylo's expert-reviewed 50-task subset of BixBench removes, clarifies, or corrects ambiguous questions and flawed ground truths. Scores are substantially higher than on the full benchmark and must not be treated as directly comparable to full-BixBench results.

Less a single benchmark than an open-source harness aggregating 166 biomedical benchmarks and 75 tools across 9 functional families, decoupling tool exposure / harness mode / context / scoring. Across 12 backbones, tool-equipped agents gain ~15 pts on average over prior SOTA on 8 representative benchmarks. Useful infrastructure to track.

Expert / olympiad-level reasoning across physics, chemistry, and biology (160 problems). Only the biology subset is in-scope for us; unsaturated and hard. Already in inspect_evals.

OpenAI's benchmark for replicating 20 ICML 2024 papers from scratch — not biology per se, but an agentic 'AI for science' boundary benchmark relevant to reproducible research workflows. Long-horizon and expensive to run. Already in inspect_evals.

Proxy measurement of hazardous knowledge; the WMDP-bio subset is the biology slice, used for unlearning research. A safety benchmark — higher is not 'better'. Include with that framing.