PatchEval (verified dockerized subset)

Evaluates LLMs on automated vulnerability repair (AVR) using 230 real-world CVEs (2015-2025) across Python, JavaScript, and Go, each packaged in a Docker sandbox. An agent inspects a vulnerable repository and submits a unified diff patch. Scoring runs fix-run.sh inside a clean evaluator sandbox, executing both security (PoC) tests and functionality (unit) tests. A patch is successful only if it passes both test types.

Maintained By

@yzlzbql

Code

patcheval

Paper

https://arxiv.org/abs/2511.11019v1

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: bytedance/PatchEval@fe6f402

Evaluates LLMs on automated vulnerability repair (AVR) using 230 real-world CVEs (2015-2025) across Python, JavaScript, and Go, each packaged in a Docker sandbox. An agent inspects a vulnerable repository and submits a unified diff patch. Scoring runs fix-run.sh inside a clean evaluator sandbox, executing both security (PoC) tests and functionality (unit) tests. A patch is successful only if it passes both test types.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/bytedance/PatchEval
cd PatchEval
git checkout fe6f402a9f54253c7ed0db0a92065898d89f94e2
uv sync

Running evaluations

CLI

uv run inspect eval src/patcheval_inspect/task.py@patcheval_verified --model openai/gpt-5-nano

Python

from inspect_ai import eval
from patcheval_inspect.task import patcheval_verified

eval(patcheval_verified(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: bytedance/PatchEval.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/patcheval_inspect/task.py@patcheval_verified --limit 10
uv run inspect eval src/patcheval_inspect/task.py@patcheval_verified --max-connections 10
uv run inspect eval src/patcheval_inspect/task.py@patcheval_verified --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗