ChipBench (Verilog Generation subset)

ChipBench evaluates LLMs on Verilog code generation from natural-language specifications across three module categories: self-contained (29 cases), non-self-contained hierarchical (6 cases), and CPU IP submodules (9 cases). Each case provides a prompt, golden Verilog, and a test harness combining directed corner cases with 1000+ random stimuli compiled via iVerilog. Scoring uses pass@1/5/10 over 20 samples at temperature=0.85, top_p=0.95, executed in a Docker sandbox.

Maintained By

@Plswearpants

Code

chipbench_verilog_gen

Paper

https://arxiv.org/abs/2601.21448

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: Plswearpants/Inspect-Eval-ChipBench@8a60e2e

ChipBench evaluates LLMs on Verilog code generation from natural-language specifications across three module categories: self-contained (29 cases), non-self-contained hierarchical (6 cases), and CPU IP submodules (9 cases). Each case provides a prompt, golden Verilog, and a test harness combining directed corner cases with 1000+ random stimuli compiled via iVerilog. Scoring uses pass@1/5/10 over 20 samples at temperature=0.85, top_p=0.95, executed in a Docker sandbox.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/Plswearpants/Inspect-Eval-ChipBench
cd Inspect-Eval-ChipBench
git checkout 8a60e2e8914139c1e73fe62ae8018dc240605ff3
uv sync

Running evaluations

CLI

uv run inspect eval src/chipbench/chipbench.py@chipbench_verilog_gen --model openai/gpt-5-nano

Python

from inspect_ai import eval
from chipbench.chipbench import chipbench_verilog_gen

eval(chipbench_verilog_gen(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: Plswearpants/Inspect-Eval-ChipBench.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/chipbench/chipbench.py@chipbench_verilog_gen --limit 10
uv run inspect eval src/chipbench/chipbench.py@chipbench_verilog_gen --max-connections 10
uv run inspect eval src/chipbench/chipbench.py@chipbench_verilog_gen --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗