PaperBench: Evaluating AI’s Ability to Replicate AI Research

Agents are evaluated on their ability to replicate 20 ICML 2024 Spotlight and Oral papers from scratch. Given a research paper PDF, an addendum with clarifications, and a rubric defining evaluation criteria, the agent must reproduce the paper’s key results by writing and executing code.

Contributed By

@vhong-aisi

Code

src/inspect_evals/paperbench

Paper

https://arxiv.org/abs/2504.01848

Overview

PaperBench evaluates AI agents on their ability to reproduce the results of machine learning research papers.

Paper: https://arxiv.org/abs/2504.01848

Overview

Agents are evaluated on their ability to replicate 20 ICML 2024 Spotlight and Oral papers from scratch. Given a research paper PDF, an addendum with clarifications, and a rubric defining evaluation criteria, the agent must reproduce the paper’s key results by writing and executing code.

Original implementation: https://github.com/openai/frontier-evals/tree/main/project/paperbench

The instructions.txt is copied from the version of openai/frontier-evals repo in December 2025

Installation

There are two ways of using Inspect Evals, from pypi as a dependency of your own project and as a standalone checked out GitHub repository.

If you are using it from pypi, install the package via:

pip install inspect_evals[paperbench]

If you are using Inspect Evals in its repository, start by installing the dependencies with:

uv sync --extra paperbench

Usage

PaperBench uses two tasks: paperbench runs the agent to reproduce papers, and paperbench_score scores the submissions in a fresh sandbox.

# Step 1: Run agent on papers (saves submissions to ./submissions/)
inspect eval inspect_evals/paperbench --model <model>

# Step 2: Score submissions in fresh sandbox
inspect eval inspect_evals/paperbench_score --model openai/o3-mini-2025-01-31 -T judge_type=simple

Both tasks use ./submissions/ by default, so they work together automatically. This two-task design ensures reproduce.sh runs in a fresh sandbox, matching the original PaperBench implementation’s approach of testing reproduction in a clean environment.

Blacklist Monitor

The paperbench task includes a blacklist URL monitor that checks whether the agent accessed forbidden resources (e.g. the paper’s original code repository). When a possible violation is detected, a warning is logged with the sample ID and the matched URL(s).

Between Step 1 and Step 2, review the task 1 logs for blacklist warnings. For each flagged sample, inspect the agent transcript to confirm whether a genuine violation occurred. Per the paper’s methodology, confirmed violations should score 0.

Deviations from the reference monitor

This monitor is a port of the OpenAI reference implementation, with three differences in how a blacklisted URL is matched:

Reference Here
Normalization applied to the blacklist entry only, matched against the raw log line both the blacklist entry and the scan line
Trailing / kept stripped
scp-like SSH URLs (git@host:path) not matched rewritten to host/path, so they are matched

The scp-like rewrite means a run can be flagged here that the reference monitor would let pass. It is deliberate: git clone is already one of the monitor’s trigger commands, every URL in the shipped blacklist.txt files is a GitHub repository, and git@github.com:owner/repo.git is the clone form GitHub’s own “Code → SSH” tab offers — so without it the monitor is bypassed by the default SSH workflow rather than by any evasive action. The deviation can only move blacklist_url_monitor from 1.0 to 0.0, never the reverse, and it does not touch the paper score on its own: that still requires a human to confirm the violation and pass the sample to skip_paper_ids.

To skip confirmed violations in scoring (saving judge compute):

inspect eval inspect_evals/paperbench_score --model openai/o3-mini-2025-01-31 \
  -T judge_type=simple \
  -T skip_paper_ids='["adaptive-pruning", "robust-clip"]'

Models

PaperBench involves multiple models at different stages:

Model Stage Purpose
Solver model paperbench task The agent that reads the paper and attempts to reproduce it by writing code
Judge model paperbench_score task Evaluates submissions against the rubric. Used for (1) ranking relevant files and (2) scoring each criterion

The --model flag sets the model for each task: the solver model for paperbench, and the judge model for paperbench_score.

Important: To compare results across different solver models, use the same judge model for scoring. The original PaperBench paper uses openai/o3-mini-2025-01-31 as the judge.

Task Options

# Run on dev split (3 papers for development/testing)
inspect eval inspect_evals/paperbench --model <model> -T split=dev

# Run on specific papers
inspect eval inspect_evals/paperbench --model <model> -T paper_ids=adaptive-pruning
inspect eval inspect_evals/paperbench --model <model> -T paper_ids='["adaptive-pruning", "robust-clip"]'

# Task-level limits
inspect eval inspect_evals/paperbench --model <model> \
  -T max_time_in_hours=12 \
  -T message_limit=50 \
  -T token_limit=1000000

# Custom submissions directory
inspect eval inspect_evals/paperbench --model <model> -T submissions_dir=./my-submissions
inspect eval inspect_evals/paperbench_score --model <model> -T submissions_dir=./my-submissions

# Solver options (configure the default solver)
inspect eval inspect_evals/paperbench --model <model> \
  --solver inspect_evals/paperbench_solver \
  -S max_attempts=3 \
  -S tool_timeout=600

# Scoring task options
inspect eval inspect_evals/paperbench_score --model <model> \
  -T reproduce_timeout_hours=12 \
  -T retry_threshold_minutes=5

# Skip specific papers from scoring (e.g. blacklist violations)
inspect eval inspect_evals/paperbench_score --model <model> \
  -T skip_paper_ids='["adaptive-pruning"]'

# Sandbox type (both tasks support this)
inspect eval inspect_evals/paperbench --model <model> -T sandbox_type=docker
inspect eval inspect_evals/paperbench_score --model <model> -T sandbox_type=docker

Splits

Split Papers Description
None 23 All available papers (default)
prod 20 Papers with results reported in the PaperBench paper
dev 3 Development papers for testing

Pre-downloading Dataset

To download all paper files to cache before running the eval:

from inspect_evals.paperbench.dataset import paperbench_dataset
paperbench_dataset()  # Downloads all 23 papers to cache

Files are cached in ~/.cache/inspect_evals/paperbench/ (Linux/macOS) by default.

Parameters

paperbench

  • paper_ids (str | list[str] | None): Paper ID(s) to reproduce (mutually exclusive with split). (default: None)
  • split (Optional[Literal['dev', 'prod']]): “dev” (3 papers), “prod” (20 papers), or None for all 23. (default: None)
  • solver (Solver | None): Optional solver to use. (default: None)
  • submissions_dir (str): Directory to save submissions. (default: './submissions')
  • max_time_in_hours (float): Time limit for the agent in hours. (default: 6.0)
  • message_limit (int | None): Maximum messages per sample (default 100, None for unlimited). (default: 100)
  • token_limit (int | None): Maximum tokens per sample (None for unlimited). (default: None)
  • sandbox_type (Literal['docker']): Sandbox type to use. (default: 'docker')
  • sandbox_config (SandboxEnvironmentSpec | None): Optional SandboxEnvironmentSpec to run against a non-default provider (e.g. k8s). If None, the bundled docker compose file is used. The sandbox here is task-level, so this is a spec rather than the per-sample callback used by the other migrated evals. (default: None)

Structure

paperbench/
├── dataset.py                 # HuggingFace dataset loader
├── images/
│   ├── Dockerfile             # Sandbox environment
│   ├── compose.yaml
│   └── agent.env.example      # Template for agent.env (API keys)
├── instructions.txt           # Agent instructions template
├── score/                     # Scoring logic
├── paperbench.py              # Inspect Task definition
└── solvers.py                 # Default agent solver

Paper files (PDF, addendum, rubric, etc.) are automatically downloaded from HuggingFace and cached locally.

Sandbox Environment

The agent runs in a Docker container with:

  • Python 3.12 (Debian-based)
  • Basic tools: git, curl, wget, build-essential, ffmpeg
  • Working directories: /home/paper (read-only inputs), /home/submission (agent output)

API Keys

To provide API keys to the agent sandbox:

  1. Copy the sample env file:

    cp src/inspect_evals/paperbench/images/agent.env.example src/inspect_evals/paperbench/images/agent.env

  2. Fill in your API keys in agent.env:

    OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_API_KEY=...
HF_TOKEN=...

The agent.env file is gitignored and will not be committed. These environment variables will be available inside the sandbox container.

Differences with Original PaperBench Implementation

Feature Our Environment Original
Docker-in-Docker Not supported Agents can build/run Docker containers in the sandbox
API Keys Copied to file and set as environment variables Only copied to file

Docker-in-Docker: The original implementation maps the docker.sock file from the host machine into the sandbox. While this allows the agent to test its reproduction script, it also allows the agent to escape the sandbox by issuing commands directly to the host’s Docker daemon. The authors might have had another layer of defense for this, but typical users don’t. Therefore, we’ve decided to leave this feature out.

Solver

The default solver (paperbench_solver) uses Inspect’s react() agent and tools for maintainability. It has similar prompts to the original paper implementation, but differ in a few ways. The table below outlines the differences.

Limits are enforced at the Task level: time_limit (default 6 hours, set from max_time_in_hours), message_limit (default 100), and token_limit (default unlimited).

Differences from Original PaperBench Implementation

Our implementation differ with the original basic_agent_plus in the following ways:

Feature Our Solver Original
Implementation Inspect’s react() agent Custom agent loop
Tools bash, python, text_editor, web_browser bash, python, read_file_chunk, search_file, web_browser
Limits Task-level (time, message, token) Custom real-time tracking with retry time subtraction
Context management truncation="auto" Message pruning (preserves 70%)
Periodic time limit and git reminders None Periodic updates every 5 steps
Model-specific None Special handling for o3/o4 and anthropic models

Our solver prioritizes simplicity and maintainability while the original optimizes for long-running evaluations with progress feedback.

Also note that the basic_iterative_agent is mostly a basic_agent_plus without a submit tool and a different continuation prompt. So, most of the above still apply.

Scoring

Scoring happens in a separate task (paperbench_score) that:

  1. Loads submissions from ./submissions/
  2. Runs reproduce.sh in a fresh sandbox
  3. Uses a judge to evaluate results against the rubric

Judge Types

Judge Type Description
dummy Returns 0 for all scores (default, for testing)
simple LLM-based judge that evaluates each rubric criterion

To use the LLM-based judge:

inspect eval inspect_evals/paperbench_score --model openai/o3-mini-2025-01-31 -T judge_type=simple

Note: The simple judge only supports OpenAI models due to its use of tiktoken for token counting.

How criteria aggregate

The rubric is a tree. Each leaf is one criterion, graded by the judge, and each parent is the weighted average of its children. Score.value carries two keys:

Key Meaning
score The weighted rubric score, over the criteria that could be graded.
graded_weight_fraction The share of the rubric weight those criteria represent. This is 1.0 when every criterion was graded.

A criterion the judge could not grade is left out of the weighted average, on both sides of the division. Each rubric leaf takes three model calls: one to rank the submission’s files, one to grade, and one to parse the grading essay into a score. Every reply is classified before use. A refusal, an empty reply, or a file ranking that names nothing readable is a judge failure rather than evidence about the submission, and is tried again: the ranking and grading calls are attempted up to twice each, and the parse up to twice per grading reply. A parser fault re-runs only the parse. A grading essay that the parser reports as carrying no score is re-graded, since re-parsing the same text cannot recover it. A criterion still ungraded after that records a grader_failure_mode of grader_refusal, grader_parse_failure, grader_schema_mismatch or grader_invalid_grade, with the attempt counts and the final grading reply’s stop_reason, in its leaf metadata. The judge failing to grade a criterion is not evidence that the reproduction did not meet it, so counting it as a zero would charge the judge’s failure to the model under test.

When no criterion in the rubric could be graded, the sample is returned unscored and appears under unscored_samples rather than as a score of zero.

Leaving ungraded criteria out of the divisor raises the reported score if the judge fails more often on criteria the reproduction did not meet. Read graded_weight_fraction alongside score for this reason. scored_samples and unscored_samples do not show this loss, because a partially graded rubric still produces a scored sample.

Score.metadata additionally records num_leaf_nodes, num_invalid_leaf_nodes and grader_failure_modes for reading individual samples. These are per-sample diagnostics and no metric reads them.

Failures of the machinery around the judge error the sample rather than producing a score: sandbox reads, a defect in the grading code, and judge API errors that survive inspect_ai’s own retries, such as an authentication failure or an unsupported parameter. A grading prompt that exceeds the judge’s context window is not an API error on the pinned inspect_ai; it comes back as a reply with stop_reason model_length. It is retried once with half the file budget and errors the sample if it overflows again, because that means this implementation’s token budgeting is wrong. Rubrics carry a few hundred criteria, so one such failure discards that sample’s grading, and re-running it with --retry-on-error re-runs reproduce.sh for the whole sample.

Deviation from the reference implementation

The aggregation above differs from openai/frontier-evals, which these scoring modules are ported from. The reference averages over every child, including those whose grading failed. Scores produced by this implementation are therefore not directly comparable with published PaperBench numbers when the judge fails to grade any criterion. graded_weight_fraction reports how much of a given run this affects, and is 1.0 when the two implementations agree.

See SCORING_DESIGN.md for the scoring architecture.

Agent Requirements

The agent must create a reproduce.sh script in /home/submission that:

  • Installs dependencies
  • Runs the reproduction code
  • Is executable via bash reproduce.sh

Evaluation Report

No results are published for this implementation. A contributor’s estimate on #334 put a full run of the 20-paper prod split at roughly $400 of solver API credits and 12 hours of GPU time per paper, before judge costs, and the maintainers have not funded one. The paper reports an average replication score of 21.0% for Claude 3.5 Sonnet (New) with the authors’ scaffold, and 4.1% for the BasicAgent gpt-4o baseline in the reference repository. Those are the reference numbers to compare against once results exist.

The solver differs from the original scaffold in the ways listed under Differences from Original PaperBench Implementation, which limits how directly a run here compares with the paper. Scoring also differs: criteria the judge could not grade are left out of the weighted average, as described under Deviation from the reference implementation. A run should record the judge model used, since the paper’s numbers were produced with openai/o3-mini-2025-01-31.

For a smoke test of the pipeline without judge spend, run the dev split with the default dummy judge:

inspect eval inspect_evals/paperbench --model <model> -T split=dev -T max_time_in_hours=0.5
inspect eval inspect_evals/paperbench_score --model <model>

Changelog

[5-C] - 2026-09-24

  • Criteria the judge could not grade are now left out of the weighted rubric average, on both sides of the division. Previously an unparseable judge reply, a reply that failed the parse schema, and a reply the parser reported as carrying no score each produced a score of 0.0 for that criterion, which was indistinguishable from a criterion the reproduction did not meet. Judge replies are now classified before use: a refusal, an empty reply, or a file ranking that names nothing readable is tried again (ranking and grading are attempted up to twice each, parsing up to twice per grading reply) and otherwise leaves the criterion ungraded with a grader_failure_mode. Previously a refused or empty grading reply was handed to the parser as if it were an essay, and a failed file ranking graded the criterion against no files, which the grading prompt scores as 0. The score mean can move in either direction: per-sample scores rise when a judge reply was ungradeable, and samples whose grader call raised now leave the denominator.
  • A rubric in which no criterion could be graded now returns Score.unscored() instead of 0.0, so the sample leaves the metric denominator and is counted under unscored_samples.
  • Failures of the machinery around the judge, such as sandbox reads, a defect in the grading code, or a judge API error that survives inspect_ai’s retries, now error the sample instead of producing a score of 0.0. A grading prompt that overflows the judge’s context window is retried once with half the file budget and errors the sample if it overflows again. Rubrics carry a few hundred criteria, so one such failure discards that sample’s grading, and --retry-on-error re-runs reproduce.sh for the whole sample.
  • Score.value changed from a single float to a dict with keys score and graded_weight_fraction, each reported as its own metric. Leaving ungraded criteria out of the divisor raises the reported score if the judge fails more often on criteria the reproduction did not meet, so graded_weight_fraction reports how much of the rubric was graded. It is 1.0 when every criterion was graded.
  • This aggregation differs from the upstream frontier-evals implementation, which averages over all criteria including those whose grading failed. See the Scoring section.

[4-C] - 2026-09-15

  • Fix the blacklist URL monitor missing scp-like SSH clones (git clone git@github.com:owner/repo.git). The host and path sit either side of a : in that spelling, so a blacklisted repository written https://github.com/owner/repo was not matched. All 21 URLs in the shipped blacklist.txt files are GitHub repositories, and this is the clone form GitHub’s own “Code → SSH” tab offers.

[3-C] - 2026-09-02

  • Added a task-level sandbox_config: SandboxEnvironmentSpec override for non-default providers (#1115).

[3-B] - 2026-05-08

  • Add blacklist URL monitor to the paperbench task that warns when an agent may have accessed forbidden resources (e.g. the paper’s original code repository).
  • Add skip_paper_ids parameter to paperbench_score so human reviewers can exclude confirmed blacklist violations from scoring per the paper’s methodology.

[2-A] - 2026-02-16

  • Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

  • Adds backoff policy for functions that connect to huggingface servers.

Static checks

Results of inspect-evals-lint at the commit this documentation was built from. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…