BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Reasoning

Reading comprehension dataset that queries for complex, non-factoid information, and require difficult entailment-like inference to solve.

Contributed By

@seddy-aisi

Overview

BoolQ is a benchmark containing natural questions that have simple yes/no answers.

Usage

First, install the inspect_ai and inspect_evals Python packages with:

pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals

Then, evaluate against one more models with:

inspect eval inspect_evals/boolq --model openai/gpt-4o

If you don’t want to specify the --model each time you run an evaluation, create a .env configuration file in your working directory that defines the INSPECT_EVAL_MODEL environment variable along with your API key. For example:

INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>

Options

You can control a variety of options from the command line. For example:

inspect eval inspect_evals/boolq --limit 10
inspect eval inspect_evals/boolq --max-connections 10
inspect eval inspect_evals/boolq --temperature 0.5

See inspect eval --help for all available options.

Dataset

BoolQ is a question answering dataset for yes/no questions containing 3,270 samples. Each sample is a triplet of question, passage, answer.

Here is an example from the dataset (after processing by Inspect):

question: is harry potter and the escape from gringotts a roller coaster ride

passage: Harry Potter and the Escape from Gringotts is an indoor steel roller coaster at Universal Studios Florida, a theme park located within the Universal Orlando Resort. Similar to dark rides, the roller coaster utilizes special effects in a controlled-lighting environment and also employs motion-based 3-D projection of both animation and live-action sequences to enhance the experience. The ride, which is themed to the Gringotts Wizarding Bank, became the flagship attraction for the expanded Wizarding World of Harry Potter when it opened on July 8, 2014

Scoring

A simple accuracy is calculated over the datapoints.