PinchBench: Skill Composition for Coding Agents

This is an Inspect AI wrapper of the original PinchBench implementation. It does not contain its own dataset or scoring logic. The original implementation can be found at https://github.com/pinchbench/skill. Users provide a pinned checkout of the original repository, tested here at commit 819384ae830492365b8363fc26bc2602e73f216d, and the wrapper reports native PinchBench results through Inspect’s scoring interface. At that commit the upstream task manifest lists 147 real-world OpenClaw agent tasks in 11 categories (productivity, research, writing, coding, analysis, CSV analysis, log analysis, meeting analysis, memory, skills, and integrations); every native task is one Inspect sample, so –limit, –sample-id and –max-samples apply. The native harness grades 25 tasks automatically, 21 with an LLM judge and 101 with both; the wrapper runs that harness in Docker, serves the judge from the same OpenAI-compatible endpoint as the agent (or a judge_model it exposes), and scores each sample as native points earned over points available, reporting Inspect mean/stderr, the upstream aggregate (total points over total available) as native_score, and per-category means. Task containers run with –network host so they can reach the per-run model proxy, and only the key configured for the benchmarked endpoint is forwarded to them. It requires Docker, an OpenAI-compatible model endpoint, the pinned Docker base image node:22-bookworm@sha256:c601a46abb4d2ab80a9dc3da208d50d1122642d53f17a101926ace71e5a9bf1c, openclaw@2026.6.10, and the exact Python dependencies pinned in the upstream pyproject.toml and Dockerfile.

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: zytoh0/pinch-wildclawbench-inspect@25961ba

This is an Inspect AI wrapper of the original PinchBench implementation. It does not contain its own dataset or scoring logic. The original implementation can be found at https://github.com/pinchbench/skill. Users provide a pinned checkout of the original repository, tested here at commit 819384ae830492365b8363fc26bc2602e73f216d, and the wrapper reports native PinchBench results through Inspect’s scoring interface. At that commit the upstream task manifest lists 147 real-world OpenClaw agent tasks in 11 categories (productivity, research, writing, coding, analysis, CSV analysis, log analysis, meeting analysis, memory, skills, and integrations); every native task is one Inspect sample, so –limit, –sample-id and –max-samples apply. The native harness grades 25 tasks automatically, 21 with an LLM judge and 101 with both; the wrapper runs that harness in Docker, serves the judge from the same OpenAI-compatible endpoint as the agent (or a judge_model it exposes), and scores each sample as native points earned over points available, reporting Inspect mean/stderr, the upstream aggregate (total points over total available) as native_score, and per-category means. Task containers run with –network host so they can reach the per-run model proxy, and only the key configured for the benchmarked endpoint is forwarded to them. It requires Docker, an OpenAI-compatible model endpoint, the pinned Docker base image node:22-bookworm@sha256:c601a46abb4d2ab80a9dc3da208d50d1122642d53f17a101926ace71e5a9bf1c, openclaw@2026.6.10, and the exact Python dependencies pinned in the upstream pyproject.toml and Dockerfile.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/zytoh0/pinch-wildclawbench-inspect
cd pinch-wildclawbench-inspect
git checkout 25961ba22c55182e12f48fc03afbdc3ab52ccba0
uv sync

Running evaluations

CLI

uv run inspect eval src/pinchbench/pinchbench.py@pinchbench --model openai/gpt-5-nano

Python

from inspect_ai import eval
from pinchbench.pinchbench import pinchbench

eval(pinchbench(), model="openai/gpt-5-nano")

View logs

Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: zytoh0/pinch-wildclawbench-inspect.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/pinchbench/pinchbench.py@pinchbench --limit 10 --sample-shuffle
uv run inspect eval src/pinchbench/pinchbench.py@pinchbench --max-connections 10
uv run inspect eval src/pinchbench/pinchbench.py@pinchbench --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗

Static checks

Results of inspect-evals-lint at the registered commit. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…