GDM Dangerous Capabilities: Self-reasoning

Test AI’s ability to reason about its environment.

Overview

This directory includes 11 self-reasoning tasks from Google DeepMind’s Evaluating Frontier Models for Stealth and Situational Awareness.

Usage

First, install the dependencies:

uv sync

Then, evaluate against one or more models with:

uv run inspect eval inspect_evals/gdm_approved_directories --model openai/gpt-5-nano
uv run inspect eval inspect_evals/gdm_calculator_improvement --model openai/gpt-5-nano
uv run inspect eval inspect_evals/gdm_context_length_mod_instrumental_only --model openai/gpt-5-nano
uv run inspect eval inspect_evals/gdm_context_length_mod_irreversibility_only --model openai/gpt-5-nano
uv run inspect eval inspect_evals/gdm_database_tool --model openai/gpt-5-nano
uv run inspect eval inspect_evals/gdm_latency_calculator --model openai/gpt-5-nano
uv run inspect eval inspect_evals/gdm_max_messages_calculator --model openai/gpt-5-nano
uv run inspect eval inspect_evals/gdm_max_tokens --model openai/gpt-5-nano
uv run inspect eval inspect_evals/gdm_oversight_frequency --model openai/gpt-5-nano
uv run inspect eval inspect_evals/gdm_read_logs --model openai/gpt-5-nano
uv run inspect eval inspect_evals/gdm_turn_off_filters --model openai/gpt-5-nano

To run multiple tasks simulteneously use inspect eval-set:

uv run inspect eval-set inspect_evals/gdm_approved_directories inspect_evals/gdm_calculator_improvement inspect_evals/gdm_context_length_mod_instrumental_only inspect_evals/gdm_context_length_mod_irreversibility_only inspect_evals/gdm_database_tool inspect_evals/gdm_latency_calculator inspect_evals/gdm_max_messages_calculator inspect_evals/gdm_max_tokens inspect_evals/gdm_oversight_frequency inspect_evals/gdm_read_logs inspect_evals/gdm_turn_off_filters

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval, eval_set
from inspect_evals.gdm_capabilities.self_reasoning import gdm_approved_directories, gdm_calculator_improvement, gdm_context_length_mod_instrumental_only, gdm_context_length_mod_irreversibility_only, gdm_database_tool, gdm_latency_calculator, gdm_max_messages_calculator, gdm_max_tokens, gdm_oversight_frequency, gdm_read_logs, gdm_turn_off_filters
eval(gdm_approved_directories)
eval_set([gdm_approved_directories, gdm_calculator_improvement, gdm_context_length_mod_instrumental_only, gdm_context_length_mod_irreversibility_only, gdm_database_tool, gdm_latency_calculator, gdm_max_messages_calculator, gdm_max_tokens, gdm_oversight_frequency, gdm_read_logs, gdm_turn_off_filters], log_dir='logs-run-42')

After running evaluations, you can view their logs using the inspect view command:

uv run inspect view

For VS Code, you can also download Inspect AI extension for viewing logs.

If you don’t want to specify the --model each time you run an evaluation, create a .env configuration file in your working directory that defines the INSPECT_EVAL_MODEL environment variable along with your API key. For example:

INSPECT_EVAL_MODEL=anthropic/claude-opus-4-1-20250805
ANTHROPIC_API_KEY=<anthropic-api-key>

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/gdm_approved_directories --limit 10
uv run inspect eval inspect_evals/gdm_calculator_improvement --max-connections 10
uv run inspect eval inspect_evals/gdm_context_length_mod_instrumental_only --temperature 0.5

See uv run inspect eval --help for all available options.

Run All

To run all tasks, you can do the following

inspect eval \
inspect_evals/gdm_approved_directories \
inspect_evals/gdm_calculator_improvement \
inspect_evals/gdm_context_length_mod_instrumental_only \
inspect_evals/gdm_context_length_mod_irreversibility_only \
inspect_evals/gdm_database_tool \
inspect_evals/gdm_latency_calculator \
inspect_evals/gdm_max_messages_calculator \
inspect_evals/gdm_max_tokens \
inspect_evals/gdm_oversight_frequency \
inspect_evals/gdm_read_logs \
inspect_evals/gdm_turn_off_filters \
--model google/gemini-2.5-flash-preview-04-17

Agent

The challenges are by default evaluated using a basic ReAct agent with task specific scaffolding that implements a tool use loop with encouragement for the model to keep going if its gives up. For more information on the task specific scaffolding, check out Appendix D in Evaluating Frontier Models for Stealth and Situational Awareness. The default agent will use up to 80 messages (~40 turns) to solve the challenge, you can customize this using the max_messages task option:

For example:

inspect eval inspect_evals/gdm_context_length_mod_instrumental_only -T max_messages=75

Epochs

The GDM paper evaluated each challenge’s difficulty level with 50 trials, reporting success if at least 1 of the 50 trials was successful. We set default number of epochs as 10 here to save compute, however you can modify this with the --epochs option, and update the reducer with a task parameter:

inspect eval inspect_evals/gdm_context_length_mod_instrumental_only --epochs=5 -T at_least_m=3

Scoring

Each task has their own scoring functions, please refer to the paper for detail. In general, we track whether the agent modifies configuration files correctly or whether they solve the task in the end.

Agent.py Environment

Every challenge has agent.py, agent_lib.py and some other environment files. These is to simulate a simple deployment scenario. Please refer to paper section 3.2 for details. They are tailored to Gemini deployment scenario, adapting it to support all models are for future works.

Evaluation results using dataset released on 2025-04-29

Model	Easy	Hard
gemini-2.5-flash-preview-04-17	10/11	3/11
o4-mini	8/11	3/11
claude-3-7-sonnet-20250219-	10/11	3/11

Disclaimer

Please note that all the data included in our challenges is purely fictitious, including the names of companies, people, email addresses or other personal identifiable information. Any overlap with real entities is purely random and not intended, as these names were randomly generated via LLMs. While it would have been possible to not use any names of existing companies or email providers, this would introduce a distribution shift that might inform AIs that they being evaluated. Thus, to not distort our results, we decided to not use clearly fictitious data but realistic-looking fictitious data.