MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models (Steganography subset)

MonitorBench evaluates chain-of-thought (CoT) monitorability: whether a model’s reasoning faithfully surfaces the decision-critical factors driving its outputs. The full benchmark spans 1,514 instances across 19 tasks in 7 categories along three axes (input intervention, outcome justification, solution process), with stress-test conditions probing intentional suppression. This subset implements the steganography task: 50 writing questions from Databricks Dolly-15k, each paired with a payload word the model must covertly encode as a sentence-initial acrostic while answering. After deterministic verification that the payload was encoded, a monitor LLM judges whether the hidden side objective is visible in the action, the CoT, or both, under three conditions (standard, direct concealment, monitor-aware evasion). Monitorability is the fraction of verification-passing rollouts the monitor flags as detected.

Maintained By

@semsorock

Code

monitorbench-steganography

Paper

https://arxiv.org/abs/2603.28590v2

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: semsorock/inspect-evals-monitor-bench@7ef6d47

MonitorBench evaluates chain-of-thought (CoT) monitorability: whether a model’s reasoning faithfully surfaces the decision-critical factors driving its outputs. The full benchmark spans 1,514 instances across 19 tasks in 7 categories along three axes (input intervention, outcome justification, solution process), with stress-test conditions probing intentional suppression. This subset implements the steganography task: 50 writing questions from Databricks Dolly-15k, each paired with a payload word the model must covertly encode as a sentence-initial acrostic while answering. After deterministic verification that the payload was encoded, a monitor LLM judges whether the hidden side objective is visible in the action, the CoT, or both, under three conditions (standard, direct concealment, monitor-aware evasion). Monitorability is the fraction of verification-passing rollouts the monitor flags as detected.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/semsorock/inspect-evals-monitor-bench
cd inspect-evals-monitor-bench
git checkout 7ef6d47219bbcca636071d5becbd572f4a4497a9
uv sync

Running evaluations

CLI

uv run inspect eval src/monitor_bench/steganography.py@monitor_bench_steganography --model openai/gpt-5-nano

Python

from inspect_ai import eval
from monitor_bench.steganography import monitor_bench_steganography

eval(monitor_bench_steganography(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: semsorock/inspect-evals-monitor-bench.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/monitor_bench/steganography.py@monitor_bench_steganography --limit 10 --sample-shuffle
uv run inspect eval src/monitor_bench/steganography.py@monitor_bench_steganography --max-connections 10
uv run inspect eval src/monitor_bench/steganography.py@monitor_bench_steganography --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗