DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation

Code generation benchmark with a thousand data science problems spanning seven Python libraries.

Overview

DS-1000 is a code generation benchmark with a thousand data science problems spanning seven Python libraries: NumPy, Pandas, TensorFlow, PyTorch, SciPy, Scikit-learn, and Matplotlib.

The benchmark was introduced in DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation, Lai et al 2022.

Dataset

DS-1000 contains realistic problems with diverse format and content, has low evaluation error rate, and defends against memorisation by modifying problems from the original source. Examples were sourced from the StackOverflow, hand picked, perturbed and reformulated to include execution context, sample data, unit tests and other evaluation criteria. Perturbations included surface changes (e.g. rephrasing), small semantic changes that did not increase difficulty of the problem (e.g. changing min to max), and large semantic changes that did increase difficulty of the problem.

Dataset is available on Hugging Face: xlangai/DS-1000.

Example

Following is an shortened example problem from the dataset:

I have the following DataFrame:
Col1 Col2 Col3 Type
0 1 2 3 1
1 4 5 6 1
...ommited for brevity...

I would like to shuffle the order of the DataFrame's rows according to a list. For example
...ommited for brevity...

How can I achieve this?

A:
<code>
import pandas as pd
import numpy as np

df = pd.DataFrame({'Col1': [1, 4, 7, 10, 13, 16],
'Col2': [2, 5, 8, 11, 14, 17],
'Col3': [3, 6, 9, 12, 15, 18],
'Type': [1, 1, 2, 2, 3, 3]})
List = np.random.permutation(len(df))
</code>
result = ... # put solution in this variable
BEGIN SOLUTION
<code>

Model is expected to complete the prompt with the code similar to:

result = df.iloc[List]
</code>

Scoring

Model solution is tested using the unit tests provided in the dataset. Tests include unit tests with sample inputs and expected outputs as well as surface-form constraints that check solution for usage of specific APIs.

Reported metrics include pass@1 accuracy which is calculated as a percentage of problems for which the model’s solution passed all unit tests.

Generated code is evaluated within a sandbox environment.

Note: if you are running this eval with agents based on the react() system, you may wish to pass this parameter submit=AgentSubmit(answer_only=True) to your react() call. This ensures that only the code inserted into submit() is tested, and not other commentary from the agent.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/ds1000 --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.ds1000 import ds1000
eval(ds1000)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/ds1000 --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/ds1000 --max-connections 10
uv run inspect eval inspect_evals/ds1000 --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

ds1000

• library (str | None): (default: None)
• perturbation (str | None): (default: None)
• shuffle (bool): (default: True)
• docker_handling (DockerHandling): (default: <DockerHandling.DEFAULT: 'default'>)

Changelog

[3-B] - 2026-08-03

• Pin sandbox image references to immutable tags/digests, identical to what latest resolved to at pin time, so registry pushes cannot silently change the evaluation environment (#2062).

[3-A] - 2026-02-16

• Migrate version to new scheme. See #907.

[2.0.0] - 2026-02-11

• Improve scorer post-processing (will remove some false negatives).

[1.0.1] - 2025-12-18

• Adds backoff policy for functions that connect to huggingface servers.