DeceptionBench: Belief-vs-Behaviour Deception under Scenario Pressure

Measures belief-vs-behaviour deception in LLMs. For each of 180 scenarios the eval elicits the model’s internal belief from a neutral prompt, then places the model under scenario pressure and captures its chain-of-thought (<think>) and final answer (<output>). Three verbatim GPT-4o judges check whether the reasoning tracks the belief (R~U_m) while the answer diverges from it (Y!~U_m). The headline metric is the Deception Tendency Rate (DTR) — the fraction of scenarios showing that deceptive pattern — reported overall and across five categories (sycophancy, strategic deception, honesty evasion, alignment faking, sandbagging).

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: WatchTree-19/inspect-deceptionbench@8683bf5

Measures belief-vs-behaviour deception in LLMs. For each of 180 scenarios the eval elicits the model’s internal belief from a neutral prompt, then places the model under scenario pressure and captures its chain-of-thought (<think>) and final answer (<output>). Three verbatim GPT-4o judges check whether the reasoning tracks the belief (R~U_m) while the answer diverges from it (Y!~U_m). The headline metric is the Deception Tendency Rate (DTR) — the fraction of scenarios showing that deceptive pattern — reported overall and across five categories (sycophancy, strategic deception, honesty evasion, alignment faking, sandbagging).

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/WatchTree-19/inspect-deceptionbench
cd inspect-deceptionbench
git checkout 8683bf5a480f938b812aa5078fb0ee7d12a74483
uv sync

Running evaluations

CLI

uv run inspect eval src/inspect_deceptionbench/deceptionbench.py@deceptionbench --model openai/gpt-5-nano

Python

from inspect_ai import eval
from inspect_deceptionbench.deceptionbench import deceptionbench

eval(deceptionbench(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: WatchTree-19/inspect-deceptionbench.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/inspect_deceptionbench/deceptionbench.py@deceptionbench --limit 10
uv run inspect eval src/inspect_deceptionbench/deceptionbench.py@deceptionbench --max-connections 10
uv run inspect eval src/inspect_deceptionbench/deceptionbench.py@deceptionbench --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗