ChipBench (Reference Model Generation variant)

ChipBench evaluates LLMs on chip design tasks using 264 test cases: 44 Verilog generation problems (self-contained, hierarchical, CPU IP modules), 89 debugging cases (arithmetic, assignment, timing, state-machine bugs, zero/one-shot), and 132 reference model generation samples (Python, SystemC, CXXRTL). Scoring uses iVerilog functional simulation against golden implementations with 1000+ random stimuli. Metrics are pass@1/5/10 across 20 samples per problem at temperature=0.85, top_p=0.95.

Maintained By

@Plswearpants

Code

chipbench_refmodel

Paper

https://arxiv.org/abs/2601.21448

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: Plswearpants/Inspect-Eval-ChipBench@8a60e2e

ChipBench evaluates LLMs on chip design tasks using 264 test cases: 44 Verilog generation problems (self-contained, hierarchical, CPU IP modules), 89 debugging cases (arithmetic, assignment, timing, state-machine bugs, zero/one-shot), and 132 reference model generation samples (Python, SystemC, CXXRTL). Scoring uses iVerilog functional simulation against golden implementations with 1000+ random stimuli. Metrics are pass@1/5/10 across 20 samples per problem at temperature=0.85, top_p=0.95.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/Plswearpants/Inspect-Eval-ChipBench
cd Inspect-Eval-ChipBench
git checkout 8a60e2e8914139c1e73fe62ae8018dc240605ff3
uv sync

Running evaluations

CLI

uv run inspect eval src/chipbench/chipbench.py@chipbench_refmodel --model openai/gpt-5-nano

Python

from inspect_ai import eval
from chipbench.chipbench import chipbench_refmodel

eval(chipbench_refmodel(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: Plswearpants/Inspect-Eval-ChipBench.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/chipbench/chipbench.py@chipbench_refmodel --limit 10
uv run inspect eval src/chipbench/chipbench.py@chipbench_refmodel --max-connections 10
uv run inspect eval src/chipbench/chipbench.py@chipbench_refmodel --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗