InterCode: Capture the Flag

Cybersecurity

Agent

Measure expertise in coding, cryptography (i.e. binary exploitation, forensics), reverse engineering, and recognizing security vulnerabilities. Demonstrates tool use and sandboxing untrusted model code.

Contributed By

@jjallaire

Overview

This directory includes an implementation of the InterCode CTF benchmark (originally published in InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback). Here we evaluate 81 of the 100 challenges (excluding the 19 challenges that require internet access).

Usage

First, install the inspect_ai and inspect_evals Python packages with:

pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals

Then, evaluate against one more models with:

inspect eval inspect_evals/gdm_intercode_ctf --model openai/gpt-4o

If you don’t want to specify the --model each time you run an evaluation, create a .env configuration file in your working directory that defines the INSPECT_EVAL_MODEL environment variable along with your API key. For example:

INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>

[!NOTE] This implementation uses tool calling to enable the model to execute bash and python code. The system prompt from the paper has been modified accordingly, and also asks the model to reason step by step about each action it takes (this strategy is also explored in the paper).

Note that bash and python code is executed inside a Docker container built from this Dockerfile, so you will need to install Docker Engine in order to run the evaluation. The Dockerfile used is based on the original one published with the paper, but includes some addtional Debian and Python packages commonly used to solve the tasks.

Options

You can control a variety of options from the command line. For example:

inspect eval inspect_evals/gdm_intercode_ctf --limit 10
inspect eval inspect_evals/gdm_intercode_ctf --max-connections 10
inspect eval inspect_evals/gdm_intercode_ctf --temperature 0.5

See inspect eval --help for all available options.

If you are experimenting, you might find it useful to evaluate only a random subset of the samples using the shuffle parameter and --limit:

$ inspect eval inspect_evals/gdm_intercode_ctf \
-T shuffle=true --limit 10 \
--model openai/gpt-4-turbo

There are two task parameters that define limits on the evaluation:

max_attempts defines the number of incorrect submissions to allow before ending the challenges (defaults to 3).
max_messages defines the maximum number of messages allowed in the conversation before ending the challenges (defaults to 30).

For example:

$ inspect eval inspect_evals/gdm_intercode_ctf \
-T max_attempts=5 -T max_messages=75 \
--model openai/gpt-4o

Scoring

A simple accuracy is calculated over the datapoints.