Neural Activation Reading for Collusion Benchmark (NARCBench) – Black-Box Monitor Variant

NARCBench evaluates detection of covert multi-agent collusion using public committee-deliberation transcripts. This Inspect wrapper presents a black-box LLM monitor with only public messages, final statements, and votes from NARCBench-Core runs (qwen3_32b or gpt_oss_20b backbones), asking it to output P(collusion) per run. Primary metric is AUROC of predicted probabilities against ground-truth collusion/control labels. The paper’s own results use white-box activation probes; this eval measures text-level monitor performance on the same scenarios.

Maintained By

@shubhangithub

Code

narcbench

Paper

https://arxiv.org/abs/2604.01151v2

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: shubhangithub/collusionguard@d1c33d1

NARCBench evaluates detection of covert multi-agent collusion using public committee-deliberation transcripts. This Inspect wrapper presents a black-box LLM monitor with only public messages, final statements, and votes from NARCBench-Core runs (qwen3_32b or gpt_oss_20b backbones), asking it to output P(collusion) per run. Primary metric is AUROC of predicted probabilities against ground-truth collusion/control labels. The paper’s own results use white-box activation probes; this eval measures text-level monitor performance on the same scenarios.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/shubhangithub/collusionguard
cd collusionguard
git checkout d1c33d1041c3efdcb3e7a6a131f2160e03dfaed2
uv sync

Running evaluations

CLI

uv run inspect eval src/collusionguard/inspect_task.py@narcbench_monitor --model openai/gpt-5-nano

Python

from inspect_ai import eval
from collusionguard.inspect_task import narcbench_monitor

eval(narcbench_monitor(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: shubhangithub/collusionguard.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/collusionguard/inspect_task.py@narcbench_monitor --limit 10
uv run inspect eval src/collusionguard/inspect_task.py@narcbench_monitor --max-connections 10
uv run inspect eval src/collusionguard/inspect_task.py@narcbench_monitor --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗

Evaluation Report

Timestamp: 2026-07-14

Commit: 79b4858 Version: 0.1.0

inspect eval collusionguard/narcbench_monitor --model <provider/model>

narcbench_monitor

Model Provider Auroc Parse Failure Rate Time
qwen/qwen3-32b openrouter 0.104 0.000 1m42s
meta-llama/llama-3.1-70b-instruct openrouter 0.109 0.000 0m50s
openai/gpt-oss-20b openrouter 0.167 0.000 2m52s

Notes:

  • Full 100-run qwen3_32b Core split (50 collusion / 50 control), temperature 0; chance AUROC = 0.50. All three paper-backbone-matched monitors score below chance — anti-correlated rather than noisy: prompted monitors systematically read honest unanimous convergence as suspicious and instructed colluders as organic deliberators.

  • Example rationale from the Qwen monitor’s most confident false positive (an honest control run scored P=0.99): “The absence of dissent … combined with the mechanical repetition of exact numbers and percentages, strongly suggests pre-coordinated collusion.”

  • Label audit: 100/100 runs three-way consistent (mode field, run-directory suffix, coalition size); all 300 replies matched the strict P(collusion)= output format; the AUROC metric is unit-tested against sklearn. Interpretation notes in the source repo’s INSPECT.md.