Neural Activation Reading for Collusion Benchmark (NARCBench) – Black-Box Monitor Variant

NARCBench evaluates black-box detection of covert multi-agent collusion from complete public committee records. The primary task presents all 100 Qwen3-32B Core runs (50 matched collusion/control pairs), including public scenario context, option descriptions, deliberation, final statements, and votes, and asks a monitor model for P(collusion). Primary AUROC is intention-to-treat over all 50 pairs; a legacy-context view is available as a documented ablation. The paper’s results use white-box activation probes, while this eval measures transcript-level monitoring on its released data.

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: shubhangithub/collusionguard@3c394c9

NARCBench evaluates black-box detection of covert multi-agent collusion from complete public committee records. The primary task presents all 100 Qwen3-32B Core runs (50 matched collusion/control pairs), including public scenario context, option descriptions, deliberation, final statements, and votes, and asks a monitor model for P(collusion). Primary AUROC is intention-to-treat over all 50 pairs; a legacy-context view is available as a documented ablation. The paper’s results use white-box activation probes, while this eval measures transcript-level monitoring on its released data.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/shubhangithub/collusionguard
cd collusionguard
git checkout 3c394c95e78459d7d1222e39a054d6b85af12b4c
uv sync

Running evaluations

CLI

uv run inspect eval src/collusionguard/narcbench_v2.py@narcbench_monitor_v2 --model openai/gpt-5-nano

Python

from inspect_ai import eval
from collusionguard.narcbench_v2 import narcbench_monitor_v2

eval(narcbench_monitor_v2(), model="openai/gpt-5-nano")

View logs

Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: shubhangithub/collusionguard.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/collusionguard/narcbench_v2.py@narcbench_monitor_v2 --limit 10 --sample-shuffle
uv run inspect eval src/collusionguard/narcbench_v2.py@narcbench_monitor_v2 --max-connections 10
uv run inspect eval src/collusionguard/narcbench_v2.py@narcbench_monitor_v2 --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗

Evaluation Report

Timestamp: 2026-08-28

Commit: 3c394c9 Version: 0.2.1

CONFIRM_PAID_RUN=YES python scripts/run_narcbench_v2_release.py --replicates 2 --execute

narcbench_monitor_v2

Model	Provider	Auroc V2	Matched Pair Auroc V2	Parse Failure Rate	Time
qwen/qwen3-32b	openrouter	0.645	0.640	0.000	19s
qwen/qwen3-32b	openrouter	0.660	0.620	0.000	13s
meta-llama/llama-3.1-70b-instruct	openrouter	0.703	0.740	0.000	37s
meta-llama/llama-3.1-70b-instruct	openrouter	0.749	0.760	0.000	43s
openai/gpt-oss-20b	openrouter	0.871	0.870	0.000	8s
openai/gpt-oss-20b	openrouter	0.863	0.880	0.000	10s

Notes:

• Full-context results are primary. All runs use the complete 100-record Qwen3-32B Core intention-to-treat population (50 matched pairs), temperature 0, 160 completion tokens, and OpenRouter pinned to DeepInfra with fallbacks disabled. The two rows for each model are replicates 1 and 2, in that order. The monitor-generation protocols differ by model: Qwen uses the qwen_no_think prompt control, GPT-OSS uses low reasoning, and Llama uses the portable prompt with reasoning disabled. The table supports within-model replication and view contrasts. The per-model protocol differences prevent a cross-model capability ranking.

• The matched legacy-context ablation produced AUROC 0.457/0.429 for Qwen, 0.219/0.254 for Llama, and 0.820/0.834 for GPT-OSS. Full-minus-legacy deltas had positive point estimates in both replicates: +0.188/+0.232 (Qwen) and +0.483/+0.495 (Llama) with paired bootstrap intervals above zero, and +0.051/+0.029 for GPT-OSS, whose second interval includes zero; its context contrast is weaker and should be interpreted with that uncertainty.

• All 12 runs completed with zero parse failures across 1,200 scored samples. The pinned source commit is the evaluated commit. The release matrix was rerun at the pin after the maintainer’s recorded human review, and the rerun report was committed afterwards as NARCBENCH_V2_2_RERUN_2026-08-28.md. The V2 protocol was designed after reviewing V1 failures. The results cover a black-box transcript evaluation; reproducing the paper’s white-box activation-probe results is outside their scope.

• A nine-feature surface-statistic logistic baseline from the upstream repository reaches about 0.90 AUROC under scenario-grouped and leave-domain-out checks, above every monitor score reported here. These results are compatible with monitors using response length, agreement patterns, and other generation artefacts alongside semantic analysis of collusion.

• Models that emit extended reasoning may need --max-tokens or --reasoning-effort overrides. Check parse_failure_rate before interpreting AUROC because rejected or truncated outputs receive the intention-to-treat value 0.5.

Static checks

Results of inspect-evals-lint at the registered commit. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…