Instruction Order Sensitivity on GSM8K (PRISM Ordering Landscape)

Measures how permuting a fixed set of six reasoning instruction modules (RESTATE, IDENTIFY, PLAN, COMPUTE, CHECK, ANSWER) affects LLM accuracy on a frozen 32-question GSM8K subset, with module wording held constant and deterministic decoding (temperature 0). By default the task samples 50 seeded random orderings (1,600 samples); -T design=exhaustive evaluates all 720 (6!) orderings. Answers are scored by canonical numeric extraction, and per-ordering metrics (worst, best, range, standard deviation) report ordering-induced accuracy variance. The eval implements the paper’s ordering-landscape protocol; the paper’s 6.3%-96.9% range was measured on a single model (Gemini 2.5 Flash-Lite) and is not a property of the eval itself.

Maintained By

@bleymambwe

Code

instruction_order_gsm8k

Paper

https://arxiv.org/abs/2608.08344v1

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: bleymambwe/PRISM@103444d

Measures how permuting a fixed set of six reasoning instruction modules (RESTATE, IDENTIFY, PLAN, COMPUTE, CHECK, ANSWER) affects LLM accuracy on a frozen 32-question GSM8K subset, with module wording held constant and deterministic decoding (temperature 0). By default the task samples 50 seeded random orderings (1,600 samples); -T design=exhaustive evaluates all 720 (6!) orderings. Answers are scored by canonical numeric extraction, and per-ordering metrics (worst, best, range, standard deviation) report ordering-induced accuracy variance. The eval implements the paper’s ordering-landscape protocol; the paper’s 6.3%-96.9% range was measured on a single model (Gemini 2.5 Flash-Lite) and is not a property of the eval itself.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/bleymambwe/PRISM
cd PRISM
git checkout 103444da69839ed5be8d1fa2d388571ae2897bc2
uv sync

Running evaluations

CLI

uv run inspect eval evals/instruction_order/task.py@instruction_order_gsm8k --model openai/gpt-5-nano

Python

from inspect_ai import eval
from evals.instruction_order.task import instruction_order_gsm8k

eval(instruction_order_gsm8k(), model="openai/gpt-5-nano")

View logs

Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: bleymambwe/PRISM.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval evals/instruction_order/task.py@instruction_order_gsm8k --limit 10 --sample-shuffle
uv run inspect eval evals/instruction_order/task.py@instruction_order_gsm8k --max-connections 10
uv run inspect eval evals/instruction_order/task.py@instruction_order_gsm8k --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗

Static checks

Results of inspect-evals-lint at the registered commit. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…