BioStudyBench: Post-Cutoff Biomedical Studies
Twenty-five biomedical study tasks evaluated with data and tools, alongside paired prior-knowledge controls.
Editor's note
Study findings from July–September 2026 are scored by reference agreement. Each condition has three repeats; uncertainty is standard deviation, not a confidence interval. Retrieval restrictions and published knowledge cutoffs reduce contamination risk but do not guarantee unseen answers.
Published results
Results are separated by evaluation version and source configuration. Scores in different sets are not a single comparable ranking.
Mean ± standard deviation over three repeats of 25 tasks. Passing measures agreement with published reference findings, not comprehensive methodological correctness. Timeouts, errors and unscored answers count as failures. Temporal and retrieval controls reduce, but do not eliminate, contamination risk.
With data and tools
| Model | Score | Variant | Source |
|---|---|---|---|
| GPT-6 Astra | 94.7% reference-agreement pass rate ± 2.3 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| Gemini 3.8 Flash | 94.7% reference-agreement pass rate ± 6.1 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| Claude Opus 5.5 | 93.3% reference-agreement pass rate ± 2.3 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| GLM 5.3 | 81.3% reference-agreement pass rate ± 6.1 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| DeepSeek V4.1 Flash | 76.0% reference-agreement pass rate ± 4.0 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| GPT-6 Luna | 74.7% reference-agreement pass rate ± 6.1 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| MiMo V2.6 Pro | 66.7% reference-agreement pass rate ± 12.9 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| Qwen3.8 27B | 56.0% reference-agreement pass rate ± 10.6 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
Prior knowledge · no data or tools
| Model | Score | Variant | Source |
|---|---|---|---|
| GPT-6 Astra | 50.7% reference-agreement pass rate ± 8.3 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| Claude Opus 5.5 | 46.7% reference-agreement pass rate ± 6.1 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| Gemini 3.8 Flash | 40.0% reference-agreement pass rate ± 10.6 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| GLM 5.3 | 26.7% reference-agreement pass rate ± 2.3 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| MiMo V2.6 Pro | 26.7% reference-agreement pass rate ± 6.1 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| DeepSeek V4.1 Flash | 25.3% reference-agreement pass rate ± 6.1 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| GPT-6 Luna | 24.0% reference-agreement pass rate ± 4.0 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |
| Qwen3.8 27B | 24.0% reference-agreement pass rate ± 0.0 pp SD | 25 tasks × 3 repeats | BioStudyBench · paper v1 Table 2; Sections 3–4 2026-10-06 |