HellaSwag: Commonsense Event Continuation

Tests models’ commonsense reasoning abilities by asking them to select the most likely next step or continuation for a given everyday situation.

Contributed By

@jjallaire

Code

src/inspect_evals/hellaswag

Paper

https://arxiv.org/abs/1905.07830

Overview

HellaSwag is a dataset for commonsense inference. The model is prompted with a sentence and the model is tasked to pick the sentence choice that is the best suited continuation.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/hellaswag --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.hellaswag import hellaswag
eval(hellaswag)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/hellaswag --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/hellaswag --max-connections 10
uv run inspect eval inspect_evals/hellaswag --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

`hellaswag`

shuffle (bool): (default: False)
split (Literal['train', 'validation', 'test']): (default: 'validation')

Dataset

Here is an example prompt from the dataset (after it has been further processed by Inspect):

Choose the most plausible continuation for the story.

Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of A,B,C,D.

A man is sitting on a roof. he

A) is using wrap to wrap a pair of skis.
B) is ripping level tiles off.
C) is holding a rubik's cube.
D) starts pulling up roofing on a roof.

The model is then expected to generate reasoning steps and provide a final answer.

Scoring

An accuracy is calculated over the datapoints.

Changelog

[2-A] - 2026-02-16

Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

Adds backoff policy for functions that connect to huggingface servers.