HellaSwag: Can a Machine Really Finish Your Sentence?

Reasoning

Evaluting commonsense natural language inference: given an event description such as “A woman sits at a piano,” a machine must select the most likely followup.

Contributed By

@jjallaire

Overview

HellaSwag is a dataset for commonsense inference. The model is prompted with a sentence and the model is tasked to pick the sentence choice that is the best suited continuation.

Usage

First, install the inspect_ai and inspect_evals Python packages with:

pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals

Then, evaluate against one more models with:

inspect eval inspect_evals/hellaswag --model openai/gpt-4o

If you don’t want to specify the --model each time you run an evaluation, create a .env configuration file in your working directory that defines the INSPECT_EVAL_MODEL environment variable along with your API key. For example:

INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>

Options

You can control a variety of options from the command line. For example:

inspect eval inspect_evals/hellaswag --limit 10
inspect eval inspect_evals/hellaswag --max-connections 10
inspect eval inspect_evals/hellaswag --temperature 0.5

See inspect eval --help for all available options.

Dataset

Here is an example prompt from the dataset (after it has been further processed by Inspect):

Choose the most plausible continuation for the story.

Answer the following multiple choice question. The entire content of your response should be of the following format: ‘ANSWER: $LETTER’ (without quotes) where LETTER is one of A,B,C,D.

A man is sitting on a roof. he

  1. is using wrap to wrap a pair of skis.
  2. is ripping level tiles off.
  3. is holding a rubik’s cube.
  4. starts pulling up roofing on a roof.

The model is then expected to generate reasoning steps and provide a final answer.

Scoring

An accuracy is calculated over the datapoints.