WildClawBench: Productivity & Safety Evaluation

This is an Inspect AI wrapper of the original WildClawBench implementation. It does not contain its own dataset or scoring logic. The original implementation can be found at https://github.com/internlm/WildClawBench. Users provide a pinned checkout of the original repository, tested here at commit 86d71447413d38f38740a021cb776f64eb396ee0, with the task data downloaded from the internlm/WildClawBench HuggingFace dataset (workspace folder, revision 75f945578aa00cbdb8f46e4d42e4f4e98f704b4f), and the wrapper reports native WildClawBench results through Inspect’s scoring interface. WildClawBench contains 60 human-authored bilingual and multimodal long-horizon agent tasks across productivity flow, code intelligence, social interaction, search and retrieval, creative synthesis, and safety alignment; every native task is one Inspect sample, so –limit, –sample-id and –max-samples apply. The native harness runs OpenClaw-style agents in Docker and grades with deterministic rule checks, environment-state auditing, and LLM/VLM judges (42 of the 60 tasks call a judge, which the wrapper serves from the same OpenAI-compatible endpoint as the agent or a judge_model it exposes); each sample’s score is the task’s native overall_score, reported with Inspect mean/stderr and per-category means, and tasks whose native grading crashes raise an error rather than scoring zero (per-metric judge fallbacks are recorded in the score explanation). Task containers run with –network host so they can reach the per-run model proxy, and only the key configured for the benchmarked endpoint is forwarded to them. It requires Docker, an OpenAI-compatible model endpoint, the pinned Docker base image node:22-bookworm@sha256:c601a46abb4d2ab80a9dc3da208d50d1122642d53f17a101926ace71e5a9bf1c, openclaw@2026.6.10, and the exact Python dependencies pinned in the upstream pyproject.toml and Dockerfile.

Maintained By

@zytoh0

Code

wildclawbench

Paper

https://arxiv.org/abs/2605.10912v1

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: zytoh0/pinch-wildclawbench-inspect@25961ba

This is an Inspect AI wrapper of the original WildClawBench implementation. It does not contain its own dataset or scoring logic. The original implementation can be found at https://github.com/internlm/WildClawBench. Users provide a pinned checkout of the original repository, tested here at commit 86d71447413d38f38740a021cb776f64eb396ee0, with the task data downloaded from the internlm/WildClawBench HuggingFace dataset (workspace folder, revision 75f945578aa00cbdb8f46e4d42e4f4e98f704b4f), and the wrapper reports native WildClawBench results through Inspect’s scoring interface. WildClawBench contains 60 human-authored bilingual and multimodal long-horizon agent tasks across productivity flow, code intelligence, social interaction, search and retrieval, creative synthesis, and safety alignment; every native task is one Inspect sample, so –limit, –sample-id and –max-samples apply. The native harness runs OpenClaw-style agents in Docker and grades with deterministic rule checks, environment-state auditing, and LLM/VLM judges (42 of the 60 tasks call a judge, which the wrapper serves from the same OpenAI-compatible endpoint as the agent or a judge_model it exposes); each sample’s score is the task’s native overall_score, reported with Inspect mean/stderr and per-category means, and tasks whose native grading crashes raise an error rather than scoring zero (per-metric judge fallbacks are recorded in the score explanation). Task containers run with –network host so they can reach the per-run model proxy, and only the key configured for the benchmarked endpoint is forwarded to them. It requires Docker, an OpenAI-compatible model endpoint, the pinned Docker base image node:22-bookworm@sha256:c601a46abb4d2ab80a9dc3da208d50d1122642d53f17a101926ace71e5a9bf1c, openclaw@2026.6.10, and the exact Python dependencies pinned in the upstream pyproject.toml and Dockerfile.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/zytoh0/pinch-wildclawbench-inspect
cd pinch-wildclawbench-inspect
git checkout 25961ba22c55182e12f48fc03afbdc3ab52ccba0
uv sync

Running evaluations

CLI

uv run inspect eval src/wildclawbench/wildclawbench.py@wildclawbench --model openai/gpt-5-nano

Python

from inspect_ai import eval
from wildclawbench.wildclawbench import wildclawbench

eval(wildclawbench(), model="openai/gpt-5-nano")

View logs

Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: zytoh0/pinch-wildclawbench-inspect.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/wildclawbench/wildclawbench.py@wildclawbench --limit 10 --sample-shuffle
uv run inspect eval src/wildclawbench/wildclawbench.py@wildclawbench --max-connections 10
uv run inspect eval src/wildclawbench/wildclawbench.py@wildclawbench --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗

Static checks

Results of inspect-evals-lint at the registered commit. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…