OpenBookQA

OpenBookQA is a 4-way multiple-choice QA benchmark of 500 test questions (5,957 total) derived from elementary science facts. Each question requires combining a core science fact from a provided 1,326-fact ‘open book’ with broader common knowledge. Questions were crowdsourced with hardness filters (IR and word-association solvers must fail), then human-verified for answerability. Scoring is accuracy over answer choices. The eval also supports an ‘open book’ variant that prepends the gold supporting fact to each question.

Maintained By

@Sammy-Dabbas

Code

openbookqa

Paper

https://arxiv.org/abs/1809.02789v1

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: Sammy-Dabbas/openbookqa-eval@52222db

OpenBookQA is a 4-way multiple-choice QA benchmark of 500 test questions (5,957 total) derived from elementary science facts. Each question requires combining a core science fact from a provided 1,326-fact ‘open book’ with broader common knowledge. Questions were crowdsourced with hardness filters (IR and word-association solvers must fail), then human-verified for answerability. Scoring is accuracy over answer choices. The eval also supports an ‘open book’ variant that prepends the gold supporting fact to each question.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/Sammy-Dabbas/openbookqa-eval
cd openbookqa-eval
git checkout 52222db933d8ec8a3bbfbcd06cd065899c829680
uv sync

Running evaluations

CLI

uv run inspect eval src/openbookqa_eval/openbookqa.py@openbookqa --model openai/gpt-5-nano

Python

from inspect_ai import eval
from openbookqa_eval.openbookqa import openbookqa

eval(openbookqa(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: Sammy-Dabbas/openbookqa-eval.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/openbookqa_eval/openbookqa.py@openbookqa --limit 10
uv run inspect eval src/openbookqa_eval/openbookqa.py@openbookqa --max-connections 10
uv run inspect eval src/openbookqa_eval/openbookqa.py@openbookqa --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗