Skip to main content

SPaRC (Spatial Pathfinding Reasoning Challenge)

SPaRC evaluates spatial and rule-based reasoning via 500 test 2D grid pathfinding puzzles (difficulty 1-5) inspired by The Witness video game. Each puzzle requires drawing a continuous, non-self-intersecting path from start to end satisfying combinatorial rules (dot collection, gap avoidance, color separation, star pairing, triangle edge-counting, polyomino fitting, ylop subtraction). Scoring is exact-match: a model’s extracted coordinate path must match one of the brute-force-verified stored solutions. Metrics include per-difficulty accuracy and invalid-path rate.

Maintained By

@lkaesberg

Code

sparc

Paper

https://arxiv.org/abs/2505.16686v2

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: lkaesberg/SPaRC@72718ac

SPaRC evaluates spatial and rule-based reasoning via 500 test 2D grid pathfinding puzzles (difficulty 1-5) inspired by The Witness video game. Each puzzle requires drawing a continuous, non-self-intersecting path from start to end satisfying combinatorial rules (dot collection, gap avoidance, color separation, star pairing, triangle edge-counting, polyomino fitting, ylop subtraction). Scoring is exact-match: a model’s extracted coordinate path must match one of the brute-force-verified stored solutions. Metrics include per-difficulty accuracy and invalid-path rate.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/lkaesberg/SPaRC
cd SPaRC
git checkout 72718acdf5b4993f3e21194ca06247bef81aadb9
uv sync

Running evaluations

CLI

uv run inspect eval sparc/tasks.py@sparc --model openai/gpt-5-nano

Python

from inspect_ai import eval
from sparc.tasks import sparc

eval(sparc(), model="openai/gpt-5-nano")

View logs

Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: lkaesberg/SPaRC.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval sparc/tasks.py@sparc --limit 10 --sample-shuffle
uv run inspect eval sparc/tasks.py@sparc --max-connections 10
uv run inspect eval sparc/tasks.py@sparc --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗

Static checks

Results of inspect-evals-lint at the registered commit. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…