MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models (Goal sandbag safety subset)

MonitorBench evaluates CoT monitorability – whether an LLM’s chain-of-thought faithfully reflects the decision-critical factors driving its outputs. It comprises 1,514 instances across 19 tasks in 7 categories (input intervention, outcome justification, solution process). A monitor LLM checks whether injected decision-critical factors appear in CoTs. The monitorability score uses a causal attribution formula comparing flagging rates against a minimum expected ratio. Two stress-test conditions (direct concealment, monitor-aware evasion) quantify how much monitorability degrades under adversarial prompting.

Maintained By

@semsorock

Code

monitor_bench-goal-sandbag-safety

Paper

https://arxiv.org/abs/2603.28590v2

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: semsorock/inspect-evals-monitor-bench@2f7d519

MonitorBench evaluates CoT monitorability – whether an LLM’s chain-of-thought faithfully reflects the decision-critical factors driving its outputs. It comprises 1,514 instances across 19 tasks in 7 categories (input intervention, outcome justification, solution process). A monitor LLM checks whether injected decision-critical factors appear in CoTs. The monitorability score uses a causal attribution formula comparing flagging rates against a minimum expected ratio. Two stress-test conditions (direct concealment, monitor-aware evasion) quantify how much monitorability degrades under adversarial prompting.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/semsorock/inspect-evals-monitor-bench
cd inspect-evals-monitor-bench
git checkout 2f7d519614533febfbd7fe4d6dcac3ce10ea7339
uv sync

Running evaluations

CLI

uv run inspect eval src/monitor_bench/tasks/goal_sandbag/safety/eval.py@goal_sandbag_safety --model openai/gpt-5-nano

Python

from inspect_ai import eval
from monitor_bench.tasks.goal_sandbag.safety.eval import goal_sandbag_safety

eval(goal_sandbag_safety(), model="openai/gpt-5-nano")

View logs

Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: semsorock/inspect-evals-monitor-bench.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/monitor_bench/tasks/goal_sandbag/safety/eval.py@goal_sandbag_safety --limit 10 --sample-shuffle
uv run inspect eval src/monitor_bench/tasks/goal_sandbag/safety/eval.py@goal_sandbag_safety --max-connections 10
uv run inspect eval src/monitor_bench/tasks/goal_sandbag/safety/eval.py@goal_sandbag_safety --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗

Static checks

Results of inspect-evals-lint at the registered commit. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…