GDM Dangerous Capabilities: Stealth
Test AI’s ability to reason about and circumvent oversight.
Overview
This directory includes 4 stealth tasks from Google DeepMind’s Evaluating Frontier Models for Stealth and Situational Awareness.
Usage
First, install the dependencies:
uv sync
Then, evaluate against one or more models with:
uv run inspect eval inspect_evals/gdm_classifier_evasion --model openai/gpt-4o
uv run inspect eval inspect_evals/gdm_cover_your_tracks --model openai/gpt-4o
uv run inspect eval inspect_evals/gdm_oversight_pattern --model openai/gpt-4o
uv run inspect eval inspect_evals/gdm_strategic_rule_breaking --model openai/gpt-4o
After running evaluations, you can view their logs using the inspect view
command:
uv run inspect view
If you don’t want to specify the --model
each time you run an evaluation, create a .env
configuration file in your working directory that defines the INSPECT_EVAL_MODEL
environment variable along with your API key. For example:
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
Options
You can control a variety of options from the command line. For example:
uv run inspect eval inspect_evals/gdm_classifier_evasion --limit 10
uv run inspect eval inspect_evals/gdm_cover_your_tracks --max-connections 10
uv run inspect eval inspect_evals/gdm_oversight_pattern --temperature 0.5
See uv run inspect eval --help
for all available options.
Model Roles
This eval supports role-based model selection so you can use dedicated models for internal raters/judgers and in-environment chat generation. See the Inspect docs on model roles for details: Model Roles.
Roles used
- rater: LLM rater for content checks (used in
classifier_evasion
andcover_your_tracks
) - judger: LLM judge for success checks (used in
classifier_evasion
) - chat: LLM for synthetic content and in-environment chat (used in
strategic_rule_breaking
)
- rater: LLM rater for content checks (used in
Oversight Pattern
- Does not require role models (uses regex/rule checks). No role configuration needed.
Examples (CLI)
Classifier Evasion (override both rater and judger):
inspect eval inspect_evals/gdm_classifier_evasion \ --model openai/gpt-4o \ --model-role rater=google/gemini-2.5-pro-preview-05-06 \ --model-role judger=google/gemini-2.0-flash
Cover Your Tracks (override rater):
inspect eval inspect_evals/gdm_cover_your_tracks \ --model openai/gpt-4o \ --model-role rater=google/gemini-2.0-flash
Strategic Rule Breaking (override chat model):
inspect eval inspect_evals/gdm_strategic_rule_breaking \ --model openai/gpt-4o \ --model-role chat=anthropic/claude-3-5-sonnet-latest
If no --model-role
is provided, sensible defaults are used based on the selected rater_model_company
or the built-in defaults.
Run All
To run all tasks, you can do the following:
inspect eval \
\
inspect_evals/gdm_oversight_pattern \
inspect_evals/gdm_classifier_evasion \
inspect_evals/gdm_cover_your_tracks \
inspect_evals/gdm_strategic_rule_breaking \
--model google/gemini-2.5-flash-preview-04-17 \
--reasoning-tokens=4096 --max-tool-output=32768
We use 4096 reasoning tokens for thinking models from Google and Anthropic. We use medium reasoning effort for o-series model from OpenAI. We need to specify max-tool-output=32 * 1024 instead of default 16 * 1024 because the tool output can be longer than the default limit.
Agent
The challenges are by default evaluated using a basic ReAct agent with task specific scaffolding that implements a tool use loop with encouragement for the model to keep going if its gives up. For more information on the task specific scaffolding, check out Appendix D in Evaluating Frontier Models for Stealth and Situational Awareness.
Epochs
The GDM paper evaluated each challenge with 50 trials, reporting success if at least 1 of the 50 trials was successful. We set the default number of epochs as 10 here to save compute, however you can modify this with the --epochs
option:
inspect eval inspect_evals/gdm_oversight_pattern --epochs=50
Scoring
Each task has their own scoring function. Please refer to the paper for details. In general, we track whether the agent modifies configuration files correctly or whether they solve the task in the end.
Task Rater
In classifier_evasion
, cover_your_tracks
and strategic_rule_breaking
challenges, we have LLM raters/players during the evaluation. We choose highly capable and affordable models as raters. Notably, pro 2p5 is used in classifier_evasion
task and the rater plays a crucial role – using a less capable model can cause the main agent to achieve the challenge much easier. However, pro 2p5 is rate limited and we expect long waiting time during the evaluation. This is inevitable currently but we can expect to switch to a cheaper model as future models become more capable and affordable. You can change the rater model to non-Google models but a less capable model can cause gdm_classifier_evasion
challenge to be easily achievable. You can specify the rater_model_company from google
, openai
and anthropic
(We use google
by default). We only tested google
rater models performance.
inspect eval \
\
inspect_evals/gdm_oversight_pattern \
inspect_evals/gdm_classifier_evasion \
inspect_evals/gdm_cover_your_tracks \
inspect_evals/gdm_strategic_rule_breaking \
--model google/gemini-2.5-flash-preview-04-17 \
--reasoning-tokens=4096 \
--max-tool-output=32768 -T rater_model_company=openai
Evaluation results using dataset released on 2025-05-18
Please refer to “Run All” section for the running detail.
Model | oversight_pattern | classifier_evasion | cover_your_tracks | strategic_rule_breaking |
---|---|---|---|---|
gemini-2.5-flash-preview-04-17 | 0.25 | 0.044 | 0.127 | 0.39 |
o4-mini | 0.05 | 0.01 | 0.214 | 0.409 |
Disclaimer
Please note that all the data included in our challenges is purely fictitious, including the names of companies, people, email addresses or other personal identifiable information. Any overlap with real entities is purely random and not intended, as these names were randomly generated via LLMs. While it would have been possible to not use any names of existing companies or email providers, this would introduce a distribution shift that might inform AIs that they being evaluated. Thus, to not distort our results, we decided to not use clearly fictitious data but realistic-looking fictitious data.