HumanEval: Python Function Generation from Instructions

Assesses how accurately language models can write correct Python functions based solely on natural-language instructions provided as docstrings.

Overview

HumanEval is a benchmark to evaluate a model’s performance on synthesizing programs from docstrings. This implementation is based on the official implementation.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/humaneval --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.humaneval import humaneval
eval(humaneval)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/humaneval --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/humaneval --max-connections 10
uv run inspect eval inspect_evals/humaneval --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

humaneval

• solver (Solver | None): The solver to use for this evaluation. Defaults to the default solver. (default: None)
• instruction_prompt (str): The prompt to prepend to the code problem. (default: '\nRead the following function signature and docstring, and fully implement\nthe function described. Your response should only contain the code for\nthis function.\n\n')
• scorer (Scorer | list[Scorer] | None): The scorer to use for this evaluation. Defaults to the default scorer. (default: None)
• sandbox (str): The sandbox to use for this evaluation. (default: 'docker')

Dataset

Here is an example prompt from the dataset:

from typing import List

def has_close_elements(numbers: List[float], threshold: float) -> bool:
   """Check if in given list of numbers, are any two numbers closer to
   each other than given threshold.
   >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
   False
   >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
   True
   """

The model is then tasked to fill in the missing code pieces in order to make this a working function.

Scoring

Once a generation is completed, the entire function, accompanied with unit tests, is run in a subprocess and is considered a success if all unit tests pass.

The benchmark uses the pass@k metric to measure functional correctness. In brief terms, this is the per problem probability of at least 1 correct sample generation given k generations. It is defined using the following expectation:

\[ \text{pass@}k := \underset{\text{Problems}}{\mathbb{E}}\left[1-\frac{{n-c}\choose{k}}{{n}\choose{k}}\right] \]

where we sample \(n \geq k\) generations to reduce variance. Note that the default in this benchmark implementation is \(n = 5\), and we evaluate \(\text{pass}@k\) for \(k \in \\{1, 2, 5\\}\).

Changelog

[2-A] - 2026-02-16

• Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

• Adds backoff policy for functions that connect to huggingface servers.