WINOGRANDE: An Adversarial Winograd Schema Challenge at Scale

Set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations.

Contributed By

@xeon27

Code

src/inspect_evals/winogrande

Paper

https://arxiv.org/abs/1907.10641

Overview

WinoGrande is a collection of 44k problems inspired by the Winograd Schema Challenge. Formulated as a fill-in-a-blank task with binary options, the goal is to choose the right option for a given sentence which requires commonsense reasoning.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/winogrande --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.winogrande import winogrande
eval(winogrande)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/winogrande --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/winogrande --max-connections 10
uv run inspect eval inspect_evals/winogrande --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

`winogrande`

dataset_name (str): (default: 'winogrande_xl')
fewshot (int): (default: 5)
fewshot_seed (int): (default: 42)
fewshot_shuffle (bool): (default: True)
shuffle (bool): (default: False)

Dataset

Here is an example from the dataset:

Sentence: He never comes to my home, but I always go to his house because the [BLANK] is smaller.

Options: home, house

The model is tasked to fill the [BLANK] with either of the two options.

Evaluation

A simple accuracy is calculated over the datapoints.

Known dataset issues

The train splits contain a few repeated records (two in most configs, four in winogrande_xl), reported at allenai/winogrande discussion #8 and still open at the pinned revision. The evaluated validation split has none in any config. The train split is only used to draw the fewshot examples after shuffling, so a repeat could at most appear twice among those examples; the port no longer deduplicates for this.

Changelog

[2-A] - 2026-02-16

Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

Adds backoff policy for functions that connect to huggingface servers.

Static checks

Results of inspect-evals-lint at the commit this documentation was built from. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…