BBH: Challenging BIG-Bench Tasks

Tests AI models on a suite of 23 challenging BIG-Bench tasks that previously proved difficult even for advanced language models to solve.

Contributed By

@JoschkaCBraun

Code

src/inspect_evals/bbh

Paper

https://arxiv.org/abs/2210.09261

Overview

BIG-Bench-Hard is a subset of 27 challenging tasks from the BIG-Bench suite, where prior language models did not outperform average human-rater performance. The benchmark tasks focus on algorithmic, commonsense, and multi-step reasoning. Evaluated models can be prompted zero-shot, 3-shot answer only or 3-shot Chain of Thought.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/bbh --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.bbh import bbh
eval(bbh)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/bbh --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/bbh --max-connections 10
uv run inspect eval inspect_evals/bbh --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

bbh

  • subset_name (str | None): Name of subset to use. See BBHDatasetRegistry.MULTIPLE_CHOICE_DATASETS, BBHDatasetRegistry.BINARY_CHOICE_DATASETS, BBHDatasetRegistry.EXACT_MATCH_DATASETS or BBHDatasetRegistry.DYCK_DATASET for valid options. (default: None)
  • prompt_type (str): Type of prompt to use. One of [“zero_shot”, “answer_only”, “chain_of_thought”]. (default: 'answer_only')

You can evaluate specific datasets and specify prompting behavior using the dataset_name and prompt_type task parameters. For example:

inspect eval inspect_evals/bbh -T "dataset_name=date_understanding"
inspect eval inspect_evals/bbh -T "prompt_type=chain_of_thought"

Dataset

BIG-Bench-Hard is a question-answering dataset comprising 27 unique tasks, most containing 250 samples, except for ‘snarks’ with 178 samples, ‘penguins_in_a_table’ with 146, and ‘causal_judgement’ with 187. In total, the benchmark includes 6,511 samples across the 27 tasks, each requiring different forms of algorithmic, commonsense and multistep reasoning to predict the correct answer.

Tasks

Here’s a list of tasks categorized by dataset type:

Multiple Choice Datasets

  1. date_understanding
  2. disambiguation_qa
  3. geometric_shapes
  4. hyperbaton
  5. logical_deduction_five_objects
  6. logical_deduction_seven_objects
  7. logical_deduction_three_objects
  8. movie_recommendation
  9. penguins_in_a_table
  10. reasoning_about_colored_objects
  11. ruin_names
  12. salient_translation_error_detection
  13. snarks
  14. temporal_sequences
  15. tracking_shuffled_objects_five_objects
  16. tracking_shuffled_objects_seven_objects
  17. tracking_shuffled_objects_three_objects

Binary Choice Datasets

  1. boolean_expressions (True, False)
  2. causal_judgement (Yes, No)
  3. formal_fallacies (valid, invalid)
  4. navigate (Yes, No)
  5. sports_understanding (yes, no)
  6. web_of_lies (Yes, No)

Open Answer Datasets

  1. multistep_arithmetic_two (integer)
  2. object_counting (natural number)
  3. word_sorting (list of words)
  4. dyck_languages (closing brackets)

Here is an example from the dataset:

Q: "It is not always easy to grasp who is consuming which products. The following argument pertains to this question:
Every infrequent user of Paul Mitchell shampoo is either a rare consumer of Nioxin shampoo or a loyal buyer of Caress soap, or both.
No regular consumer of Lush soap is a rare consumer of Nioxin shampoo and, in the same time, a loyal buyer of Caress soap.
It follows that whoever is an infrequent user of Paul Mitchell shampoo is not a regular consumer of Lush soap."
Is the argument, given the explicitly stated premises, deductively valid or invalid?
Options:
- valid
- invalid

The model is required to choose the correct answer from the given options.

Scoring

A simple accuracy is calculated over the samples.

Changelog

[2-A] - 2026-02-16

  • Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

  • Adds backoff policy for functions that connect to huggingface servers.