ChipBench (Verilog Debugging subset)

ChipBench evaluates LLMs on AI-aided chip design across three tasks: Verilog generation (44 modules), Verilog debugging (89 cases), and reference model generation (132 cases in Python/SystemC/CXXRTL). The debugging task injects four bug types (arithmetic, assignment, timing, state machine) into golden Verilog modules; models receive buggy code plus a description (zero-shot) or additionally a VCD waveform (one-shot). Scoring uses iVerilog simulation against a golden reference, reporting pass@1/5/10 over 20 samples.

Maintained By

@Plswearpants

Code

chipbench_debug

Paper

https://arxiv.org/abs/2601.21448

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: Plswearpants/Inspect-Eval-ChipBench@8a60e2e

ChipBench evaluates LLMs on AI-aided chip design across three tasks: Verilog generation (44 modules), Verilog debugging (89 cases), and reference model generation (132 cases in Python/SystemC/CXXRTL). The debugging task injects four bug types (arithmetic, assignment, timing, state machine) into golden Verilog modules; models receive buggy code plus a description (zero-shot) or additionally a VCD waveform (one-shot). Scoring uses iVerilog simulation against a golden reference, reporting pass@1/5/10 over 20 samples.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/Plswearpants/Inspect-Eval-ChipBench
cd Inspect-Eval-ChipBench
git checkout 8a60e2e8914139c1e73fe62ae8018dc240605ff3
uv sync

Running evaluations

CLI

uv run inspect eval src/chipbench/chipbench.py@chipbench_debug --model openai/gpt-5-nano

Python

from inspect_ai import eval
from chipbench.chipbench import chipbench_debug

eval(chipbench_debug(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: Plswearpants/Inspect-Eval-ChipBench.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/chipbench/chipbench.py@chipbench_debug --limit 10
uv run inspect eval src/chipbench/chipbench.py@chipbench_debug --max-connections 10
uv run inspect eval src/chipbench/chipbench.py@chipbench_debug --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗