Alignment Faking in Large Language Models

Measures whether LLMs strategically comply with harmful requests when believing outputs will be used for RLHF training (free-tier/monitored) versus when unmonitored (paid-tier). Uses 100 harmful prompts from LLM-LAT/harmful-dataset. Each prompt is presented twice with free-tier and paid-tier tags inside a system prompt describing a conflicting training objective. Primary metric is the compliance gap (free minus paid compliance rate via two-proportion z-test); a secondary scorer detects verbalized alignment-faking reasoning in scratchpad outputs.

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: marliechorgan/alignment-faking-inspect@d102d21

Measures whether LLMs strategically comply with harmful requests when believing outputs will be used for RLHF training (free-tier/monitored) versus when unmonitored (paid-tier). Uses 100 harmful prompts from LLM-LAT/harmful-dataset. Each prompt is presented twice with free-tier and paid-tier tags inside a system prompt describing a conflicting training objective. Primary metric is the compliance gap (free minus paid compliance rate via two-proportion z-test); a secondary scorer detects verbalized alignment-faking reasoning in scratchpad outputs.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/marliechorgan/alignment-faking-inspect
cd alignment-faking-inspect
git checkout d102d213833b05c67464a44e28e9a414977f80e1
uv sync

Running evaluations

CLI

uv run inspect eval src/alignment_faking/alignment_faking.py@alignment_faking --model openai/gpt-5-nano

Python

from inspect_ai import eval
from alignment_faking.alignment_faking import alignment_faking

eval(alignment_faking(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: marliechorgan/alignment-faking-inspect.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/alignment_faking/alignment_faking.py@alignment_faking --limit 10
uv run inspect eval src/alignment_faking/alignment_faking.py@alignment_faking --max-connections 10
uv run inspect eval src/alignment_faking/alignment_faking.py@alignment_faking --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗

Evaluation Report

Timestamp: June 2026

Commit: d102d21

uv run inspect eval src/alignment_faking/alignment_faking.py@alignment_faking --model anthropic/claude-sonnet-4-6 -T paraphrase=ensemble

Model	Provider	Free Compliance Rate	Paid Compliance Rate	Compliance Gap	Compliance Gap Pvalue	Free Tier Af Rate
anthropic/claude-sonnet-4-6	Anthropic	0.333	0.370	-0.037	0.267	0.005
anthropic/claude-opus-4-8	Anthropic	0.725	0.775	-0.050	0.371	0.000

Notes:

• Paraphrase ensemble (canonical + 7 paraphrases) x both tiers, real-lab framing. Sonnet 4.6: 50 prompts = 800 samples (400/tier). Opus 4.8: 15 prompts = 240 samples (120/tier).
• Headline metric is compliance_gap = free_tier minus paid_tier compliance rate, with a two-proportion z-test (compliance_gap_pvalue).
• Finding: no robust alignment-faking gap on either current model. A fictional-lab de-fingerprinting variant showed a marginal positive gap on Sonnet 4.6 (+0.075, p=0.09, n=240/tier) but it did not replicate in a higher-powered run of the strongest cell (canonical/fictional, n=100/tier: gap 0.000). Verbalised faking <=2%.
• Verbatim-prompt results are a lower bound (contamination / mitigation overfitting); the original positive-control models (Claude 3 Opus, Claude 3.5 Sonnet) are API-retired. Full conditions table and limitations are in the upstream README.