ExploitBench: Capability Ladder Benchmark for LLM Cybersecurity Agents (V8 exploit development)

ExploitBench evaluates LLM agents on developing memory-corruption exploits against 41 real V8 JavaScript engine N-day vulnerabilities with all production mitigations enabled. It decomposes exploitation into 16 binary capability flags across 5 tiers (coverage, bug triggering, V8-internal primitives, general primitives, arbitrary code execution), verified by deterministic challenge-response oracles compiled into a patched d8 binary. Episodes are capped at 300 turns; best-of-N seeds are aggregated via bitmap union.

Maintained By

@ChaoticCooties

Code

exploitbench

Paper

https://arxiv.org/abs/2605.14153v1

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: ChaoticCooties/exploitbench-eval@ccdd13a

ExploitBench evaluates LLM agents on developing memory-corruption exploits against 41 real V8 JavaScript engine N-day vulnerabilities with all production mitigations enabled. It decomposes exploitation into 16 binary capability flags across 5 tiers (coverage, bug triggering, V8-internal primitives, general primitives, arbitrary code execution), verified by deterministic challenge-response oracles compiled into a patched d8 binary. Episodes are capped at 300 turns; best-of-N seeds are aggregated via bitmap union.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/ChaoticCooties/exploitbench-eval
cd exploitbench-eval
git checkout ccdd13a128cf4dd25c720e8bf1f13d2867b6ba23
uv sync

Running evaluations

CLI

uv run inspect eval src/exploitbench/exploitbench.py@eb_v8 --model openai/gpt-5-nano

Python

from inspect_ai import eval
from exploitbench.exploitbench import eb_v8

eval(eb_v8(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: ChaoticCooties/exploitbench-eval.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/exploitbench/exploitbench.py@eb_v8 --limit 10
uv run inspect eval src/exploitbench/exploitbench.py@eb_v8 --max-connections 10
uv run inspect eval src/exploitbench/exploitbench.py@eb_v8 --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗