Evals by capability

Practical single-cell RNA-seq analysis tasks — directly in the dbverse / spatial-omics wheelhouse and a high-value target for a bio-specialist leaderboard. Eight frontier models evaluated; Claude Opus 4.6 leads at 52.8% accuracy and no model clears 53%.

146 verifiable problems across five spatial technologies and seven task categories, from the scBench team. Each gives a data snapshot just before an analysis step plus a deterministic grader for recovering the key biological result. Accuracy is low (20–38% across families) and very sensitive to harness design — squarely in the spatial-omics wheelhouse. Seven frontier models evaluated; Claude Opus 4.5 leads at 38.4%.

24 evaluations spanning CosMx / Visium / Xenium / MERFISH / Slide-seq / histology / lineage data, graded deterministically over controlled vocabularies with chokepoint rubrics. Extremely hard: three model–harness pairs tie for best at just 11.1% (8/72). Directly dbverse-adjacent. Scores are per model–harness pair (harness in the variant).

21 evaluations across five research areas (melanoma CD8 reactivity, RNA+ATAC regulatory inference, human–monkey chimera development, KRAS lung-tumor aging, COVID-19 lung pathology), deterministically graded with trajectory rubrics. Brutally unsaturated: across 1,068 trajectories the best model–harness pair passes just 25.4% (16/63). Your specialty. Scores are per model–harness pair (harness in the variant).

Open-answer agentic bioinformatics: agents must run multi-step analyses over real datasets. Scores are low and the open-answer vs multiple-choice gap is large, so treat MCQ numbers with skepticism. Best public agents still miss roughly half the questions.

End-to-end bioinformatics pipelines (RNA-seq, variant calling, metagenomics) with a robustness twist: stress-tests agents under corrupted inputs and decoy files, LLM-graded for pipeline progress and outcome validity. Finds frontier agents complete pipelines without heavy scaffolding but break under perturbation. ICML 2026.

OpenAI's June 2026 benchmark for scientific 'research taste': 129 synthetic genomics / quantitative-biology / translational-medicine problems, each pairing a noisy dataset with a target estimand, graded deterministically from a known causal structure. Extremely hard and unsaturated.

OpenAI's rubric-graded evaluation of medical/health capabilities across realistic healthcare conversations. Model-graded against physician-written rubrics, so scoring cost is nontrivial. Already in inspect_evals.

Clinician-authored visual QA over radiology images — a multimodal medical benchmark. Tests vision-language models, not text-only. Already in inspect_evals.

Molecular-biology reasoning with a hard split; reports accuracy on the human-solved subset. Newer and unsaturated — a useful difficulty signal for frontier bio reasoning.

Broad chemistry knowledge and reasoning over 2,786 QA pairs. Chemistry-adjacent to biology (drug discovery, molecular properties); included as a boundary domain. Already in inspect_evals.

Broad biology-research QA (literature QA, protocols, figure reading, sequence manipulation). Already implemented in inspect_evals, so it is the natural first target to reproduce ourselves with inspect_ai. Verified per-model scores pending our own runs.

FutureHouse's successor to LAB-Bench: ~1,900 tasks in more realistic research contexts, and markedly harder — model accuracy drops 26–46% across subtasks vs the original. Supersedes LAB-Bench (v1) as the current version.

Biomedical yes/no/maybe QA over PubMed abstracts. Largely saturated by frontier models, so most useful as a sanity/regression check than a frontier signal. Already in inspect_evals.

OpenAI's 750 expert-authored tasks spanning 7 scientific workflows × 7 life-science domains, each with an expert rubric — built to capture the ambiguity and judgment calls that knowledge-QA benchmarks miss (complex artifacts, situational ambiguity, open-ended answers). Far from saturated: no model passes 22.8% of tasks, and 34.8% have a best-model pass rate under 20%. Score shown is the problem-weighted normalized rubric score; top pass rate is 36.1% (GPT-Rosalind).

Less a single benchmark than an open-source harness aggregating 166 biomedical benchmarks and 75 tools across 9 functional families, decoupling tool exposure / harness mode / context / scoring. Across 12 backbones, tool-equipped agents gain ~15 pts on average over prior SOTA on 8 representative benchmarks. Useful infrastructure to track.

Expert / olympiad-level reasoning across physics, chemistry, and biology (160 problems). Only the biology subset is in-scope for us; unsaturated and hard. Already in inspect_evals.

OpenAI's benchmark for replicating 20 ICML 2024 papers from scratch — not biology per se, but an agentic 'AI for science' boundary benchmark relevant to reproducible research workflows. Long-horizon and expensive to run. Already in inspect_evals.

Proxy measurement of hazardous knowledge; the WMDP-bio subset is the biology slice, used for unlearning research. A safety benchmark — higher is not 'better'. Include with that framing.