University of Pittsburgh

PubMedQA: A Dataset for Biomedical Research Question Answering

Biomedical question answering (QA) dataset collected from PubMed abstracts.

biomedicalliterature-qatext
Status
cataloged
Saturation
saturated
Runnable
yes
Cost to run
low

Editor's note

Biomedical yes/no/maybe QA over PubMed abstracts. Largely saturated by frontier models, so most useful as a sanity/regression check than a frontier signal. Already in inspect_evals.

Results

Verified scores pending — this eval is cataloged; our own inspect_ai reproduction is the next step.

Reproduce: inspect_evals implementation

Sources