PinchBench: Skill Composition for Coding Agents

This is an Inspect AI wrapper of the original PinchBench implementation. It does not contain its own dataset or scoring logic. The original implementation can be found at https://github.com/pinchbench/skill. Users provide a pinned checkout of the original repository, tested here at commit 819384ae830492365b8363fc26bc2602e73f216d, and the wrapper reports native PinchBench results through Inspect’s scoring interface. PinchBench contains 53 real-world OpenClaw agent tasks across productivity, research, writing, coding, analysis, email, memory, and skill-discovery categories. The native harness grades tasks automatically, with an LLM judge, or both; this wrapper runs that harness in Docker and reports the native aggregate score, or the mean of per-task numeric scores when no aggregate is present, with Inspect mean/stderr metrics. It requires Docker, an OpenAI-compatible model endpoint, the pinned Docker base image node:22-bookworm@sha256:c601a46abb4d2ab80a9dc3da208d50d1122642d53f17a101926ace71e5a9bf1c, openclaw@2026.6.10, and the exact Python dependencies pinned in the upstream pyproject.toml and Dockerfile.

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: zytoh0/pinch-wildclawbench-inspect@c7d2216

This is an Inspect AI wrapper of the original PinchBench implementation. It does not contain its own dataset or scoring logic. The original implementation can be found at https://github.com/pinchbench/skill. Users provide a pinned checkout of the original repository, tested here at commit 819384ae830492365b8363fc26bc2602e73f216d, and the wrapper reports native PinchBench results through Inspect’s scoring interface. PinchBench contains 53 real-world OpenClaw agent tasks across productivity, research, writing, coding, analysis, email, memory, and skill-discovery categories. The native harness grades tasks automatically, with an LLM judge, or both; this wrapper runs that harness in Docker and reports the native aggregate score, or the mean of per-task numeric scores when no aggregate is present, with Inspect mean/stderr metrics. It requires Docker, an OpenAI-compatible model endpoint, the pinned Docker base image node:22-bookworm@sha256:c601a46abb4d2ab80a9dc3da208d50d1122642d53f17a101926ace71e5a9bf1c, openclaw@2026.6.10, and the exact Python dependencies pinned in the upstream pyproject.toml and Dockerfile.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/zytoh0/pinch-wildclawbench-inspect
cd pinch-wildclawbench-inspect
git checkout c7d22166667c38e4a831183045b65091dcc7120a
uv sync

Running evaluations

CLI

uv run inspect eval src/pinchbench/pinchbench.py@pinchbench --model openai/gpt-5-nano

Python

from inspect_ai import eval
from pinchbench.pinchbench import pinchbench

eval(pinchbench(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: zytoh0/pinch-wildclawbench-inspect.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/pinchbench/pinchbench.py@pinchbench --limit 10
uv run inspect eval src/pinchbench/pinchbench.py@pinchbench --max-connections 10
uv run inspect eval src/pinchbench/pinchbench.py@pinchbench --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗