BBH: Challenging BIG-Bench Tasks

Tests AI models on a suite of 23 challenging BIG-Bench tasks that previously proved difficult even for advanced language models to solve.

Contributed By

@JoschkaCBraun

Code

src/inspect_evals/bbh

Paper

https://arxiv.org/abs/2210.09261

Overview

BIG-Bench-Hard is a subset of 27 challenging tasks from the BIG-Bench suite, where prior language models did not outperform average human-rater performance. The benchmark tasks focus on algorithmic, commonsense, and multi-step reasoning. Evaluated models can be prompted zero-shot, 3-shot answer only or 3-shot Chain of Thought.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/bbh --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.bbh import bbh
eval(bbh)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/bbh --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/bbh --max-connections 10
uv run inspect eval inspect_evals/bbh --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

bbh

  • subset_name (str | None): Name of subset to use. See BBHDatasetRegistry.MULTIPLE_CHOICE_DATASETS, BBHDatasetRegistry.BINARY_CHOICE_DATASETS, BBHDatasetRegistry.EXACT_MATCH_DATASETS or BBHDatasetRegistry.DYCK_DATASET for valid options. When None, all 27 subsets are evaluated. (default: None)
  • prompt_type (str): Type of prompt to use. One of [“zero_shot”, “answer_only”, “chain_of_thought”]. (default: 'answer_only')

You can evaluate specific datasets and specify prompting behavior using the subset_name and prompt_type task parameters. For example:

inspect eval inspect_evals/bbh -T "subset_name=date_understanding"
inspect eval inspect_evals/bbh -T "prompt_type=chain_of_thought"

Dataset

BIG-Bench-Hard is a question-answering dataset comprising 27 unique tasks, most containing 250 samples, except for ‘snarks’ with 178 samples, ‘penguins_in_a_table’ with 146, and ‘causal_judgement’ with 187. In total, the benchmark includes 6,511 samples across the 27 tasks, each requiring different forms of algorithmic, commonsense and multistep reasoning to predict the correct answer.

Tasks

Here’s a list of tasks categorized by dataset type:

Multiple Choice Datasets

  1. date_understanding
  2. disambiguation_qa
  3. geometric_shapes
  4. hyperbaton
  5. logical_deduction_five_objects
  6. logical_deduction_seven_objects
  7. logical_deduction_three_objects
  8. movie_recommendation
  9. penguins_in_a_table
  10. reasoning_about_colored_objects
  11. ruin_names
  12. salient_translation_error_detection
  13. snarks
  14. temporal_sequences
  15. tracking_shuffled_objects_five_objects
  16. tracking_shuffled_objects_seven_objects
  17. tracking_shuffled_objects_three_objects

Binary Choice Datasets

  1. boolean_expressions (True, False)
  2. causal_judgement (Yes, No)
  3. formal_fallacies (valid, invalid)
  4. navigate (Yes, No)
  5. sports_understanding (yes, no)
  6. web_of_lies (Yes, No)

Open Answer Datasets

  1. multistep_arithmetic_two (integer)
  2. object_counting (natural number)
  3. word_sorting (list of words)
  4. dyck_languages (closing brackets)

Here is an example from the dataset:

Q: "It is not always easy to grasp who is consuming which products. The following argument pertains to this question:
Every infrequent user of Paul Mitchell shampoo is either a rare consumer of Nioxin shampoo or a loyal buyer of Caress soap, or both.
No regular consumer of Lush soap is a rare consumer of Nioxin shampoo and, in the same time, a loyal buyer of Caress soap.
It follows that whoever is an infrequent user of Paul Mitchell shampoo is not a regular consumer of Lush soap."
Is the argument, given the explicitly stated premises, deductively valid or invalid?
Options:
- valid
- invalid

The model is required to choose the correct answer from the given options.

Scoring

A simple accuracy is calculated over the samples.

Known dataset issues

  • Two ruin_names items, 'earth, wind, & fire' (row 100 of 250) and 'rita, sue and bob too' (row 145), have their options split on the commas inside the candidate names, so the lettered options are fragments and the target is a full name that is not among them. They cannot be scored by option letter and are excluded; ruin_names runs 248 items. Reported at suzgunmirac/BIG-Bench-Hard#19; the same rows are in the Joschka/big_bench_hard mirror this port loads.
  • ruin_names row 41, 'frank kappa', has the unchanged input as its gold answer, which contradicts the task definition (suzgunmirac/BIG-Bench-Hard#17, #18). The row is kept as published.

Changelog

[3-A] - 2026-09-18

  • Binary-choice subsets, scored with answer(pattern="word"): a completion that does not match the answer pattern now scores INCORRECT with reason="invalid_response_format" rather than NOANSWER, following answer() upstream in inspect_ai#4629, which attributes an unmatched completion to the model under test. Multiple-choice (choice()) and exact-match subsets are unaffected. Headline metrics are unaffected: value_to_float maps both NOANSWER and INCORRECT to 0.0, so accuracy and stderr are unchanged. The non-answer moves from the score value to Score.reason; log analysis that counts NOANSWER values should read reason instead. See #2314.

[2-A] - 2026-02-16

  • Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

  • Adds backoff policy for functions that connect to huggingface servers.

Static checks

Results of inspect-evals-lint at the commit this documentation was built from. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…