CVEBench: Benchmark for AI Agents Ability to Exploit Real-World Web Application Vulnerabilities

Cybersecurity

inspect-evals

Agent

Characterises an AI Agent’s capability to exploit real-world web application vulnerabilities. Aims to provide a realistic evaluation of an agent’s security reasoning capability using 40 real-world CVEs.

Contributed By

Code

src/inspect_evals/cve_bench

Paper

https://arxiv.org/abs/2503.17332

Overview

CVEBench characterises an AI Agent’s ability to exploit real-world web application vulnerabilities.

This evals supports running CVEBench in Docker and Kubernetes sandboxes.

This eval branched from the original repo at d13ffc, and then added K8s support on top. There have been subsequent changes in the original.

The Kubernetes sandbox only supports a subset of CVEs. See K8S_SUPPORTED_CVE in cve_bench.py for the supported CVEs.

Prerequisites

Docker for running with -T sandbox_type=docker.
A running Kubernetes cluster that conforms to the Inspect K8s sandbox documentation for running with -T sandbox_type=k8s (the default). See the setup guides for a local cluster or a remote cluster.

We recommend running on an amd64 machine. It is not guaranteed that all challenges will work on arm64 machines.

Usage

Note: This eval has dependencies that conflict with other evals in this repo. It ships its own isolated environment under packages/cve_bench/ rather than sharing the repo’s main one.

Installation

From the repository (recommended)

Sync the eval’s environment from its package directory:

cd packages/cve_bench
uv sync

Then run evaluations from that directory:

uv run inspect eval inspect_evals/cve_bench --model openai/gpt-5-nano

Into an existing environment

If you already have a Python environment, install the eval’s package directly — it brings in inspect-evals alongside the eval-specific dependencies:

pip install /path/to/inspect_evals/packages/cve_bench/

Then run without uv run:

inspect eval inspect_evals/cve_bench --model openai/gpt-5-nano

You can also import the task directly from Python:

from inspect_ai import eval
from inspect_evals.cve_bench import cve_bench
eval(cve_bench)

From a Jupyter notebook (using Inspect Flow)

Inspect Flow manages the isolated environment for you. Install it once:

pip install inspect-flow

Then from a notebook cell, point Flow at this eval’s pyproject.toml so it builds the isolated venv from the same dependency set as uv sync:

from inspect_flow import FlowSpec, FlowTask, FlowDependencies
from inspect_flow.api import run

spec = FlowSpec(
    execution_type="venv",
    log_dir="logs/",
    dependencies=FlowDependencies(
        dependency_file="path/to/inspect_evals/packages/cve_bench/pyproject.toml",
    ),
    tasks=[FlowTask(name="inspect_evals/cve_bench", model="openai/gpt-5-nano")],
)

run(spec)  # blocking; builds isolated venv on first run, cached thereafter

After the run completes, read and analyse logs in the same notebook — the eval’s heavy dependencies are not required for log reading.

Running evaluations

After installing via any of the methods above, you can evaluate models. The examples below assume the repository approach; drop uv run if you are managing dependencies yourself.

uv run inspect eval inspect_evals/cve_bench --model openai/gpt-5-nano

After running evaluations, you can view their logs using the inspect view command:

uv run inspect view

For VS Code, you can also download the Inspect AI extension for viewing logs.

If you don’t want to specify the --model each time you run an evaluation, create a .env configuration file in your working directory that defines the INSPECT_EVAL_MODEL environment variable along with your API key. For example:

INSPECT_EVAL_MODEL=anthropic/claude-opus-4-1-20250805
ANTHROPIC_API_KEY=<anthropic-api-key>

Parameters

`cve_bench`

challenges (str | list[str] | None): The CVEs that the eval will run on. If None, will run all CVEs. (default: <K8S_SUPPORTED_CVE.CVE_2023_37999: 'CVE-2023-37999'>)
variants (str | list[str]): ‘one_day’ or ‘zero_day’. Variant of the prompt to use for the CVEs. ‘one_day’ hints about the specific exploit that can be used to exploit the web application. (default: <Variant.ONE_DAY: 'one_day'>)
agent (Solver | None): Agent framework to run against CVE Bench. Must be compatible with the default agent on CVE Bench. (default: None)
max_messages (int): The max messages allowed before the eval run terminates (for a given) CVE. (default: 50)
sandbox_type (Sandbox): ‘k8s’ or ‘docker’. the eval sandbox that the CVE is run against. For K8s, the remote or local cluster must be configured in a specific way; see the inspect k8s sandbox docs for details: https://k8s-sandbox.aisi.org.uk/getting-started/local-cluster/ (default: <Sandbox.K8S: 'k8s'>)

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/cve_bench --limit 10
uv run inspect eval inspect_evals/cve_bench --max-connections 10
uv run inspect eval inspect_evals/cve_bench --temperature 0.5

See uv run inspect eval --help for all available options.

Example invocations

Running with the default CVE:

uv run \
    inspect eval inspect_evals/cve_bench \
    --limit 1 \
    --model openai/gpt-4o \
    -T sandbox_type=k8s \
    -T variants=one_day \
    -T max_messages=100

Running with all CVEs (challenges=) using both prompt variants:

uv run \
    inspect eval inspect_evals/cve_bench \
    --model openai/gpt-4o \
    -T sandbox_type=k8s \
    -T challenges= \
    -T variants=one_day,zero_day \
    -T max_messages=100

Running with a specific CVE on docker:

uv run \
    inspect eval inspect_evals/cve_bench \
    --model openai/gpt-4o \
    -T sandbox_type=docker \
    -T challenges=CVE-2024-3495 \
    -T variants=one_day \
    -T max_messages=100

Changelog

[2-A] - 2026-02-16

Migrate version to new scheme. See #907.