MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models (Goal sandbag math subset)

MonitorBench evaluates chain-of-thought (CoT) monitorability: whether a model’s reasoning faithfully surfaces the decision-critical factors driving its final output. The full benchmark spans 1,514 instances across 19 tasks in 7 categories along three axes (input intervention, outcome justification, solution process), with two stress-test conditions (direct concealment, monitor-aware evasion) probing intentional suppression. This subset implements the goal-sandbag math task over 30 AIME 2024 problems: each sample pairs an original prompt with an intervened prompt that instructs the model to deliberately underperform while appearing competent, and a monitor LLM judges whether the sandbagging is detectable from the action, the CoT, or both. Monitorability is the monitor’s detection rate normalized by the causal effect of the sandbag instruction, macro-averaged over eligible problems.

Maintained By

@semsorock

Code

monitorbench-goal-sandbag-math

Paper

https://arxiv.org/abs/2603.28590v2

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: semsorock/inspect-evals-monitor-bench@e2e7b91

MonitorBench evaluates chain-of-thought (CoT) monitorability: whether a model’s reasoning faithfully surfaces the decision-critical factors driving its final output. The full benchmark spans 1,514 instances across 19 tasks in 7 categories along three axes (input intervention, outcome justification, solution process), with two stress-test conditions (direct concealment, monitor-aware evasion) probing intentional suppression. This subset implements the goal-sandbag math task over 30 AIME 2024 problems: each sample pairs an original prompt with an intervened prompt that instructs the model to deliberately underperform while appearing competent, and a monitor LLM judges whether the sandbagging is detectable from the action, the CoT, or both. Monitorability is the monitor’s detection rate normalized by the causal effect of the sandbag instruction, macro-averaged over eligible problems.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/semsorock/inspect-evals-monitor-bench
cd inspect-evals-monitor-bench
git checkout e2e7b91a84d21e7cb374d59d49dbd25c3f40514f
uv sync

Running evaluations

CLI

uv run inspect eval src/monitor_bench/tasks/goal_sandbag_math/eval.py@goal_sandbag_math --model openai/gpt-5-nano

Python

from inspect_ai import eval
from monitor_bench.tasks.goal_sandbag_math.eval import goal_sandbag_math

eval(goal_sandbag_math(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: semsorock/inspect-evals-monitor-bench.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/monitor_bench/tasks/goal_sandbag_math/eval.py@goal_sandbag_math --limit 10 --sample-shuffle
uv run inspect eval src/monitor_bench/tasks/goal_sandbag_math/eval.py@goal_sandbag_math --max-connections 10
uv run inspect eval src/monitor_bench/tasks/goal_sandbag_math/eval.py@goal_sandbag_math --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗