DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation

Code generation benchmark with a thousand data science problems spanning seven Python libraries.

Overview

DS-1000 is a code generation benchmark with a thousand data science problems spanning seven Python libraries: NumPy, Pandas, TensorFlow, PyTorch, SciPy, Scikit-learn, and Matplotlib.

The benchmark was introduced in DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation, Lai et al 2022.

Dataset

DS-1000 contains realistic problems with diverse format and content, has low evaluation error rate, and defends against memorisation by modifying problems from the original source. Examples were sourced from the StackOverflow, hand picked, perturbed and reformulated to include execution context, sample data, unit tests and other evaluation criteria. Perturbations included surface changes (e.g. rephrasing), small semantic changes that did not increase difficulty of the problem (e.g. changing min to max), and large semantic changes that did increase difficulty of the problem.

Dataset is available on Hugging Face: xlangai/DS-1000.

Example

Following is an shortened example problem from the dataset:

I have the following DataFrame:
Col1 Col2 Col3 Type
0 1 2 3 1
1 4 5 6 1
...ommited for brevity...

I would like to shuffle the order of the DataFrame's rows according to a list. For example
...ommited for brevity...

How can I achieve this?

A:
<code>
import pandas as pd
import numpy as np

df = pd.DataFrame({'Col1': [1, 4, 7, 10, 13, 16],
'Col2': [2, 5, 8, 11, 14, 17],
'Col3': [3, 6, 9, 12, 15, 18],
'Type': [1, 1, 2, 2, 3, 3]})
List = np.random.permutation(len(df))
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>

Model is expected to complete the prompt with the code similar to:

result = df.iloc[List]
</code>

Scoring

Model solution is tested using the unit tests provided in the dataset. Tests include unit tests with sample inputs and expected outputs as well as surface-form constraints that check solution for usage of specific APIs.

Reported metrics include pass@1 accuracy which is calculated as a percentage of problems for which the model’s solution passed all unit tests.

Generated code is evaluated within a sandbox environment.

Checking the sandbox image

ds1000_sandbox_check is a maintenance task, not a benchmark. It runs the eval’s scorer over four fixture solutions with known verdicts inside the pinned image: a correct numpy solution, a wrong one, an unparseable one and a correct pandas solution. A sample passes only if the verdict matches and every package pinned in docker-requirements.txt is installed at that version on Python 3.10, so accuracy 1.0 certifies the image for ds1000. No model is called. Run it after changing the Dockerfile or requirements, or when re-pinning the image in compose.yaml:

uv run inspect eval inspect_evals/ds1000_sandbox_check --model mockllm/model
uv run inspect eval inspect_evals/ds1000_sandbox_check --model mockllm/model -T docker_handling=build_locally

tests/ds1000/test_sandbox_check.py runs the same check under the slow and docker test gates. The helper behind it, inspect_evals.utils.sandbox_check, is shared with other image-backed evals.

Note: if you are running this eval with agents based on the react() system, you may wish to pass this parameter submit=AgentSubmit(answer_only=True) to your react() call. This ensures that only the code inserted into submit() is tested, and not other commentary from the agent.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/ds1000 --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.ds1000 import ds1000
eval(ds1000)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Maintenance tasks

The tasks below score the evaluation harness rather than the model, for example a judge meta-evaluation or a sandbox image check. Their results are not benchmark results.

• ds1000_sandbox_check: Runs the real scorer over fixture solutions with known verdicts and checks the pinned packages are installed; accuracy 1.0 certifies the sandbox image.

uv run inspect eval inspect_evals/ds1000_sandbox_check

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/ds1000 --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/ds1000 --max-connections 10
uv run inspect eval inspect_evals/ds1000 --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

ds1000

• library (str | None): (default: None)
• perturbation (str | None): (default: None)
• shuffle (bool): (default: True)
• docker_handling (DockerHandling): (default: <DockerHandling.DEFAULT: 'default'>)

ds1000_sandbox_check (maintenance task)

• docker_handling (DockerHandling): How to obtain the sandbox image, as for ds1000 (default: <DockerHandling.DEFAULT: 'default'>)

Changelog

[3-B] - 2026-08-03

• Pin sandbox image references to immutable tags/digests, identical to what latest resolved to at pin time, so registry pushes cannot silently change the evaluation environment (#2062).

[3-A] - 2026-02-16

• Migrate version to new scheme. See #907.

[2.0.0] - 2026-02-11

• Improve scorer post-processing (will remove some false negatives).

[1.0.1] - 2025-12-18

• Adds backoff policy for functions that connect to huggingface servers.

Static checks

Results of inspect-evals-lint at the commit this documentation was built from. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…