Friedrich Schiller University Jena

ChemBench: Are large language models superhuman chemists?

ChemBench is designed to reveal limitations of current frontier models for use in the chemical sciences. It consists of 2786 question-answer pairs compiled from diverse sources. Our corpus measures reasoning, knowledge and intuition across a large fraction of the topics taught in

chemistrydrug-discovery-adjacenttext
Status
cataloged
Saturation
partial
Reproduction
not yet
Published results
0 models

Editor's note

Broad chemistry knowledge and reasoning over 2,786 QA pairs. Chemistry-adjacent to biology (drug discovery, molecular properties); included as a boundary domain. Already in inspect_evals.

Published results

No source-reported scores have been added to BioInspect yet.

Reproduce: inspect_evals implementation

Sources