ComputeEval: CUDA Code Generation Benchmark

Evaluates LLM capability to generate correct CUDA code for kernel implementation, memory management, and parallel algorithm optimization tasks.

Contributed By

@Vitamoon

Code

src/inspect_evals/compute_eval

Overview

ComputeEval is a benchmark for evaluating whether LLMs can generate correct CUDA code. It is developed by NVIDIA Research and available at github.com/NVIDIA/compute-eval and huggingface.co/datasets/nvidia/compute-eval.

As noted in the original repo:

This repository executes machine-generated CUDA C++ code. While it’s unlikely that the code is malicious, it could still pose potential risks. We strongly recommend using a sandbox environment (e.g., a Docker container or virtual machine) when running generated code to minimize security risks.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/compute_eval --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.compute_eval import compute_eval
eval(compute_eval)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/compute_eval --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/compute_eval --max-connections 10
uv run inspect eval inspect_evals/compute_eval --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

compute_eval

  • limit (int | None): Maximum number of samples to evaluate (None for all) (default: None)
  • dataset_version (str): Dataset version to use. Options: “default” (latest), “2025-1”, “2025-2”, “2025-3”. Default is “default” which uses the latest version. (default: 'default')
  • skip_cuda_check (bool): Skip host CUDA environment validation. Ignored when sandbox_type is set; the host check is always skipped when using a sandbox since nvcc need not be present on the host. (default: False)
  • num_epochs (int): Number of solutions to generate per problem for pass@k evaluation. Use higher values (e.g., 10) with epoch_reducer=“pass_at_1” for unbiased pass@k estimation. (default: 1)
  • epoch_reducer (str | list[str]): Strategy to reduce multiple epoch scores. Options: “mean”, “max”, “median”, “mode”, or “pass_at_k” where k <= num_epochs. Can also be a list of reducers. (default: 'mean')
  • temperature (float): Temperature for generation. Use higher values (e.g., 0.8) when generating multiple samples for pass@k. (default: 1.0)
  • benchmark_performance (bool): If True, measure execution time of passing solutions by running them multiple times and recording the average. (default: False)
  • benchmark_runs (int): Number of times to run each passing solution when benchmark_performance is enabled, used to compute average execution time. (default: 3)
  • sandbox_type (Optional[Literal['docker']]): Sandbox type for isolating code execution. Use “docker” (default) to run code inside a container. Set to None to run directly on the host machine. (default: 'docker')

Dataset

ComputeEval contains CUDA programming challenges spanning kernel implementation, memory management, and algorithm optimization. Each task provides:

  • A natural language prompt describing the task
  • Context files (headers, interfaces) visible to the model
  • Hidden test files for evaluation
  • Build commands for compilation

Problem types:

  • cuda_cpp - C++ CUDA kernel implementations
  • cuda_python - Python CUDA code (e.g., CuPy, Numba). Currently, the dataset by NVIDIA does not contain any Python questions.

Releases:

Version Problems Status
2025.1 127 Released
2025.2 232 Released
2025.3 406 Released

Note: default is an alias for the latest version (2025.3).

Scoring

The scorer compiles the generated CUDA code using nvcc and runs the test harness to verify functional correctness. A solution is marked correct if:

  1. Compilation succeeds without errors
  2. All test cases pass (exit code 0)

Requirements

  • NVIDIA GPU with CUDA Toolkit 12.0+
  • nvcc compiler available in PATH

Other Notes

CUDA Environment Validation

Before running the benchmark, the implementation validates your CUDA environment:

  1. Checks that nvcc is available in PATH
  2. Verifies CUDA version is 12.0+ (required for -arch=native support)
  3. Tests that -arch=native compilation flag works

If validation fails, the benchmark will raise a RuntimeError with instructions. You can skip this check with skip_cuda_check=True, but compilation may fail on individual problems.

Per-Problem CUDA Version Requirements

Some problems in the dataset require specific minimum CUDA versions (specified in min_cuda_toolkit field). If a problem requires a newer CUDA version than installed, it will be skipped and marked as incorrect with metadata {"skipped": True, "reason": "cuda_version_insufficient"}.

Pass@k Evaluation

For unbiased pass@k estimation, use multiple epochs with temperature > 0:

uv run inspect eval inspect_evals/compute_eval \
    --model openai/gpt-5-nano \
    -T num_epochs=10 \
    -T epoch_reducer=pass_at_1 \
    -T temperature=0.8

This generates 10 solutions per problem and computes the probability of at least 1 correct solution.

Prompt Fidelity

This implementation uses prompts that match NVIDIA’s original implementation to ensure comparable benchmark results. The system prompt and user prompt format are taken directly from NVIDIA’s compute-eval repository.

Source Reference Validation

Some problems include source_references metadata specifying required API calls or symbols that must appear in the solution. The scorer validates these references and marks solutions as incorrect if required symbols are missing, even if they would otherwise compile and pass tests.

Performance Benchmarking

Performance benchmarking is disabled by default because it re-runs each passing solution benchmark_runs times (default 3), which can significantly increase total evaluation time. Enable it with benchmark_performance=True:

uv run inspect eval inspect_evals/compute_eval -T benchmark_performance=True -T benchmark_runs=5

This runs the solution multiple times and records average execution time in the score metadata. If the problem includes a reference_time_ms, speedup relative to the reference is also computed.

Evaluation Report

See the ComputeEval page for evaluation methodology. This report was produced using version 1-A of the repository.

Used the following command (with the model varying for each row)

uv run inspect eval inspect_evals/compute_eval --model openai/gpt-5-nano-2025-08-07
Model Accuracy Stderr Time Tokens
openai/gpt-5-nano-2025-08-07 45.3% ±0.025 45:49 2,819,584
openai/gpt-5.2-2025-12-11 60.3% ±0.024 35:58 766,827

Note: No paper or leaderboard available for comparison.