GPQA: A Graduate-Level Google-Proof Q&A Benchmark

Knowledge

Challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry (experts at PhD level in the corresponding domains reach 65% accuracy).

Contributed By

@jjallaire

Overview

GPQA is an evaluation dataset consisting of graduate-level multiple-choice questions in subdomains of physics, chemistry, and biology.

This implementation is based on simple-eval’s implementation. This script evaluates on the GPQA-Diamond subset.

Usage

First, install the inspect_ai and inspect_evals Python packages with:

pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals

Then, evaluate against one more models with:

inspect eval inspect_evals/gpqa_diamond --model openai/gpt-4o

If you don’t want to specify the --model each time you run an evaluation, create a .env configuration file in your working directory that defines the INSPECT_EVAL_MODEL environment variable along with your API key. For example:

INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>

Options

You can control a variety of options from the command line. For example:

inspect eval inspect_evals/gpqa_diamond --limit 10
inspect eval inspect_evals/gpqa_diamond --max-connections 10
inspect eval inspect_evals/gpqa_diamond --temperature 0.5

See inspect eval --help for all available options.

Dataset

Here is an example prompt from the dataset (after it has been further processed by Inspect):

Answer the following multiple choice question. The entire content of your response should be of the following format: ‘ANSWER: $LETTER’ (without quotes) where LETTER is one of A,B,C,D.

Two quantum states with energies E1 and E2 have a lifetime of 10^-9 sec and 10^-8 sec, respectively. We want to clearly distinguish these two energy levels. Which one of the following options could be their energy difference so that they can be clearly resolved?

  1. 10^-4 eV
  2. 10^-11 eV
  3. 10^-8 eV
  4. 10^-9 eV

The model is then tasked to pick the correct answer choice.

Scoring

A simple accuracy is calculated over the datapoints.