ChemBench: Are large language models superhuman chemists?

ChemBench is designed to reveal limitations of current frontier models for use in the chemical sciences. It consists of 2786 question-answer pairs compiled from diverse sources. Our corpus measures reasoning, knowledge and intuition across a large fraction of the topics taught in undergraduate and graduate chemistry curricula. It can be used to evaluate any system that can return text (i.e., including tool-augmented systems).

Overview

ChemBench is designed to reveal limitations of current frontier models for use in the chemical sciences. It consists of 2786 question-answer pairs compiled from diverse sources. Our corpus measures reasoning, knowledge and intuition across a large fraction of the topics taught in undergraduate and graduate chemistry curricula. It can be used to evaluate any system that can return text (i.e., including tool-augmented systems).

Usage

First, install the inspect_ai and inspect_evals Python packages with:

pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals

Or, if developing on a clone of the inspect_evals repo, you can install the package in editable mode with:

pip install -e ".[dev]"

Then, evaluate against one or more models on a task with:

inspect eval inspect_evals/chembench --model openai/gpt-4o -T task_name=inorganic_chemistry

The valid task are: “analytical_chemistry”, “chemical_preference”, “general_chemistry”, “inorganic_chemistry”, “materials_science”, “organic_chemistry”, “physical_chemistry”, “technical_chemistry”, “toxicity_and_safety”.

If no task name specified, the full dataset will be evaluated:

inspect eval inspect_evals/chembench --model openai/gpt-4o

Optional cot parameter can be provided with True to enable Chain-of-Thought:

inspect eval src/inspect_evals/chembench -T task_name=inorganic_chemistry -T cot=True 

After running evaluations, you can view their logs using the inspect view command:

inspect view

If you don’t want to specify the --model each time you run an evaluation, create a .env configuration file in your working directory that defines the INSPECT_EVAL_MODEL environment variable along with your API key. For example:

INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>

Options

You can control a variety of options from the command line. For example:

inspect eval inspect_evals/chembench --limit 10 -T task_name=inorganic_chemistry
inspect eval inspect_evals/chembench --max-connections 10 -T task_name=inorganic_chemistry
inspect eval inspect_evals/chembench --temperature 0.5 -T task_name=inorganic_chemistry

See inspect eval --help for all available options.

Dataset

ChenBench includes 9 tasks consisting of both multiple-choice and open-ended questions, totaling 2,786 samples. (Note: The original paper reports 2,788 samples, but the version on Hugging Face has two duplicates removed. See this issue for details.)

You can download the dataset from Hugging Face.

Scoring

pattern scorers with customized regexes are used to match the answer. Accuracy and stderr are calculated over each task and all samples.