Evaluation Configuration

Creating an evaluation using Inspect Cyber involves writing two configuration files.

Configuration File	Name	Description
Evaluation	eval.yaml or eval.yml	Details of the evaluation (see Evaluations).
Sandbox	Depends on the sandbox type (see Sandboxes).	Specifications of the sandbox environment, including its services and how they’re networked together.

The following sections describe the contents of each file in more detail.

Evaluations

Attribute	Description	Required
name	The name of the evaluation. This will be represented in the Inspect sample metadata as “eval_name”.	•
sandbox	The sandbox environment to use for the evaluation. Any sandbox environment compatible with Inspect can be used.
files	A dictionary of files to be copied into the sandbox environment. The keys are the destination file paths and the values are the source file paths relative to the evaluation configuration file. By default, files are copied into the default (agent) sandbox environment, but they can optionally include a prefix indicating a different sandbox environment into which they should be copied (e.g., victim:/root/flag.txt).
setup	Path to a shell script to be executed within the default service after files have been copied into the environment.
flag	The correct flag for the evaluation, used to grade the agent’s submission.
metadata	Metadata about the evaluation. This is copied directly into the sample metadata.
variants	A dictionary of variants. Keys are added to the sample metadata as “variant_name”. Values are described in the following section. An evaluation must have at least one variant.	•

Warning

Files are copied before the setup script is run but after sandbox native events, like Docker entrypoints. Avoid referencing files in Docker entrypoints or writing to files which will be overwritten by the evaluation configuration file. Use the setup script instead if needed.

Variants

Variants allow for the creation of multiple samples for a single evaluation. Each variant is converted into a Sample. The attributes of a variant are provided below. Note that all of the attributes other than name overlap with the attributes of an evaluation. By default, variants inherit the attribute values of the evaluation to which they belong. However, if attributes are set at the variant level, they override the ones set at the evaluation level.

In theory, this allows for combining vastly different samples under the same evaluation. In practice, it is usually recommended to only group variants into the same evaluation if they share some attributes. For example, variants are often useful for experimenting with different prompts for the same evaluation. They are also useful for supporting the same evaluation on multiple sandbox types (e.g., Docker and Kubernetes) or making minor changes to a sandbox environment (e.g., adding protections to a vulnerable service). It is up to the user to decide how to organise their evaluations and variants.

Attribute	Description	Required
prompt	Prompt to provide as the first user message to the agent.	•
sandbox	The sandbox environment to use for the variant. Any sandbox environment compatible with Inspect can be used.
files	(Same as above.)
setup	(Same as above.)
flag	The correct flag for the variant, used to grade the agent’s submission.
metadata	Metadata about the variant. This is copied directly into the sample metadata. Overlapping fields with evaluation metadata are overwritten.

Note

Although sandbox is not required at the evaluation or variant level, at least one of them must have it. In other words, every variant needs a sandbox. So if it is not provided at the variant level and there is none to inherit from the evaluation, an error will be thrown.

Sandboxes

The essence of an agentic cyber eval is the environment in which the agent is placed. Sandboxing is the process of setting up that environment.

Sandboxes have two attributes.

Attribute	Description	Required
type	The type of the sandbox environment.	•
config	Path to a config file for the sandbox environment.	•

There are multiple sandbox environment types: Inspect natively supports Docker; other developers have implemented support for Kubernetes and Proxmox; and users who wish to use other environment types can implement support for them.

The description of the sandbox environment, including its services and how they’re networked together, will be in the config file. Inspect Cyber is agnostic to the name and kind of the sandbox configuration file; it simply requires a path to it. Configuration files for Docker are typically named compose.yaml and ones for Kubernetes are usually helm-values.yaml, though this it isn’t required to follow this practice. A Docker sandbox configuration file for a basic evaluation involving only two services – the agent and a victim server – might look something like this:

compose.yaml

services:
  default:
    image: ubuntu:latest
    command: tail -f /dev/null
    init: true
  victim:
    image: victim:1.0.0
    init: true

1

The default service is the agent.

2

This keeps the container running.

3

This enables the container to respond to shutdown requests.

4

This would be a Docker container that’s already been built and tagged (e.g., docker build -t victim:1.0.0 ../images/victim).