WildClawBench: Productivity & Safety Evaluation

This is an Inspect AI wrapper of the original WildClawBench implementation. It does not contain its own dataset or scoring logic. The original implementation can be found at https://github.com/internlm/WildClawBench. Users provide a pinned checkout of the original repository, tested here at commit 86d71447413d38f38740a021cb776f64eb396ee0, and the wrapper reports native WildClawBench results through Inspect’s scoring interface. WildClawBench contains 60 human-authored bilingual and multimodal long-horizon agent tasks across productivity flow, code intelligence, social interaction, search and retrieval, creative synthesis, and safety alignment. The native harness runs OpenClaw-style agents in Docker and grades with deterministic rule checks, environment-state auditing, and LLM/VLM judges; this wrapper parses native summary*.json or per-task score.json outputs and reports the mean native overall_score across scored tasks with Inspect mean/stderr metrics. It requires Docker, an OpenAI-compatible model endpoint, the pinned Docker base image node:22-bookworm@sha256:c601a46abb4d2ab80a9dc3da208d50d1122642d53f17a101926ace71e5a9bf1c, openclaw@2026.6.10, and the exact Python dependencies pinned in the upstream pyproject.toml and Dockerfile.

Maintained By

@zytoh0

Code

wildclawbench

Paper

https://arxiv.org/abs/2605.10912v1

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: zytoh0/pinch-wildclawbench-inspect@c7d2216

This is an Inspect AI wrapper of the original WildClawBench implementation. It does not contain its own dataset or scoring logic. The original implementation can be found at https://github.com/internlm/WildClawBench. Users provide a pinned checkout of the original repository, tested here at commit 86d71447413d38f38740a021cb776f64eb396ee0, and the wrapper reports native WildClawBench results through Inspect’s scoring interface. WildClawBench contains 60 human-authored bilingual and multimodal long-horizon agent tasks across productivity flow, code intelligence, social interaction, search and retrieval, creative synthesis, and safety alignment. The native harness runs OpenClaw-style agents in Docker and grades with deterministic rule checks, environment-state auditing, and LLM/VLM judges; this wrapper parses native summary*.json or per-task score.json outputs and reports the mean native overall_score across scored tasks with Inspect mean/stderr metrics. It requires Docker, an OpenAI-compatible model endpoint, the pinned Docker base image node:22-bookworm@sha256:c601a46abb4d2ab80a9dc3da208d50d1122642d53f17a101926ace71e5a9bf1c, openclaw@2026.6.10, and the exact Python dependencies pinned in the upstream pyproject.toml and Dockerfile.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/zytoh0/pinch-wildclawbench-inspect
cd pinch-wildclawbench-inspect
git checkout c7d22166667c38e4a831183045b65091dcc7120a
uv sync

Running evaluations

CLI

uv run inspect eval src/wildclawbench/wildclawbench.py@wildclawbench --model openai/gpt-5-nano

Python

from inspect_ai import eval
from wildclawbench.wildclawbench import wildclawbench

eval(wildclawbench(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: zytoh0/pinch-wildclawbench-inspect.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/wildclawbench/wildclawbench.py@wildclawbench --limit 10
uv run inspect eval src/wildclawbench/wildclawbench.py@wildclawbench --max-connections 10
uv run inspect eval src/wildclawbench/wildclawbench.py@wildclawbench --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗