Friedrich Schiller University Jena

ChemBench: Are large language models superhuman chemists?

ChemBench is designed to reveal limitations of current frontier models for use in the chemical sciences. It consists of 2786 question-answer pairs compiled from diverse sources. Our corpus measures reasoning, knowledge and intuition across a large fraction of the topics taught in

chemistrydrug-discovery-adjacenttext
Status
cataloged
Saturation
partial
Runnable
yes
Cost to run
medium

Editor's note

Broad chemistry knowledge and reasoning over 2,786 QA pairs. Chemistry-adjacent to biology (drug discovery, molecular properties); included as a boundary domain. Already in inspect_evals.

Results

Verified scores pending — this eval is cataloged; our own inspect_ai reproduction is the next step.

Reproduce: inspect_evals implementation

Sources