DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

Evaluates reading comprehension where models must resolve references in a question, perhaps to multiple input positions, and perform discrete operations over them (such as addition, counting, or sorting).

Overview

DROP is a crowdsourced, adversarially-created, 96k reading comprehension benchmark, in which a system must resolve references in a question, perhaps to multiple input positions, and perform discrete operations over them.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/drop --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.drop import drop
eval(drop)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/drop --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/drop --max-connections 10
uv run inspect eval inspect_evals/drop --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

drop

• fewshot (int): Number of few shot examples to use. (default: 3)
• fewshot_seed (int): Random seed for sampling few shot examples. (default: 42)
• fewshot_shuffle (bool): Whether or not to randomly sample the fewshot examples. (default: True)

Dataset

The DROP dataset contains 9,535 samples. Here is an example from the dataset:

Passage:  The Eagles began their season at Bank of America Stadium for a Week 1 duel with the Carolina Panthers.  Philadelphia trailed early in the first quarter as Panthers running back DeAngelo Williams ran 11 yards for a Carolina touchdown on their first drive.  The Eagles answered with a 49-yard field goal from kicker David Akers. In the second quarter, Philadelphia exploded with points as defensive end Victor Abiamiri returned a fumble 2 yards for a touchdown, wide receiver DeSean Jackson returned a punt 85 yards for a touchdown, and quarterback Donovan McNabb completed a 9-yard TD pass to tight end Brent Celek and a 4-yard touchdown pass to running back Brian Westbrook. Carolina ended the period with kicker John Kasay booting a 22-yard field goal. In the third quarter, the Eagles closed out their scoring with McNabb scoring on a 3-yard touchdown run.  However, he was hit late by several Carolina tacklers who cracked his ribs on the right side, knocking him out of the game. Kevin Kolb came in for McNabb and closed out the game for the victorious Eagles.

Question: How many yards was the longest scoring play of the first quarter?

The model is tasked to answer the question by referring to multiple parts in the passage. The prompts are based on OpenAI’s simple-evals.

Scoring

A simple accuracy is calculated over the datapoints.

Changelog

[2-A] - 2026-02-16

• Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

• Adds backoff policy for functions that connect to huggingface servers.