PIQA: Physical Commonsense Reasoning Test

Measures the model’s ability to apply practical, everyday commonsense reasoning about physical objects and scenarios through simple decision-making questions.

Contributed By

@seddy-aisi

Code

src/inspect_evals/piqa

Paper

https://arxiv.org/abs/1911.11641

Overview

PIQA is a benchmark to measure the model’s physical commonsense reasoning.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/piqa --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.piqa import piqa
eval(piqa)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/piqa --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/piqa --max-connections 10
uv run inspect eval inspect_evals/piqa --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

`piqa`

No task parameters.

Dataset

Here is an example prompt from the dataset (after it has been further processed by Inspect):

The entire content of your response should be of the following format: 'ANSWER:\n$LETTER' (without quotes) where LETTER is one of A,B.

Given either a question or a statement followed by two possible solutions labelled A and B, choose the most appropriate solution. If a question is given, the solutions answer the question. If a statement is given, the solutions explain how to achieve the statement.

How do I ready a guinea pig cage for it's new occupants?

A) Provide the guinea pig with a cage full of a few inches of bedding made of ripped paper strips, you will also need to supply it with a water bottle and a food dish.  
B) Provide the guinea pig with a cage full of a few inches of bedding made of ripped jeans material, you will also need to supply it with a water bottle and a food dish.

The model is then tasked to pick the correct choice.

Scoring

A simple accuracy is calculated over the datapoints.

Changelog

[2-A] - 2026-02-16

Migrate version to new scheme. See #907.