SciCode: A Research Coding Benchmark Curated by Scientists

SciCode tests the ability of language models to generate code to solve scientific research problems. It assesses models on 65 problems from mathematics, physics, chemistry, biology, and materials science.

Overview

SciCode tests the ability of language models to generate code to solve scientific research problems. This eval assesses models on 65 problems from mathematics, physics, chemistry, biology, and materials science.

A SciCode main problem is composed of a sequence of subproblems. Subproblems are intended to be solved sequentially—their solution code may depend upon the solution code of their predecessors.

This implementation is based on the original implementation and paper preprint.

Prerequisites

Install Docker, per the installation guide.

Usage

Installation

This eval requires extra dependencies.

pip install inspect-evals[scicode]

From a checkout of this repository:

uv sync --extra scicode

Running evaluations

uv run inspect eval inspect_evals/scicode --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.scicode import scicode
eval(scicode)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Parameters

scicode

• provide_scientific_background (bool): (default: False)
• timeout (int): (default: 300)
• include_dev_set (bool): (default: False)

Optional Flags

• Provide Scientific Background: supply models with relevant background information, compiled by scientists.

inspect eval inspect_evals/scicode --model openai/gpt-4o -T provide_scientific_background=True

• Timeout: set a custom timeout interval, in seconds. By default, attempts at code execution time out after 5 minutes.

inspect eval inspect_evals/scicode --model openai/gpt-4o -T timeout=600

• Include dev set: include 15 extra SciCode problems that were reserved as a dev set. Caveat lector: these are on average easier problems. Including them will result in a noticeably higher success rate than using the main problem set alone. Furthermore, they were released alongside their solution code, so are more at risk of data contamination.

inspect eval inspect_evals/scicode --model openai/gpt-4o -T include_dev_set=True

Evaluation

Once complete, the subproblem solution provided by the model is executed in a Docker container along with test cases, and is considered correct if all test cases pass. A main problem is regarded as correct if all its subproblems are scored as correct.

The code is taken from the model’s response by extract_code (util.py), using the shared inspect_evals.utils.code helpers. Of the python-labelled code blocks (or unlabelled blocks, if there is no labelled one), it uses the last one that parses and defines the function named in the subproblem’s function header, so a corrected solution written after a draft is used rather than the draft, and a trailing usage example or fragment is skipped. If no block defines it, the first block is used, as in the original SciCode harness (extract_python_script), which differs only in also removing import lines. If deleting only the fence markers gives code that parses and defines a function or class the chosen block lacks, that code is used instead, and a response with no fence is used as-is with any stray fence markers removed.

Example Problem

Problem ID

33

Main problem description

Generate an array of Chern numbers for the Haldane model on a hexagonal lattice by sweeping the following parameters: the on-site energy to next-nearest-neighbor coupling constant ratio (\(m/t_2\)) and the phase (\(\\phi\)) values. Given the lattice spacing \(a\), the nearest-neighbor coupling constant \(t_1\), the next-nearest-neighbor coupling constant \(t_2\), the grid size \(\\delta\) for discretizing the Brillouin zone in the \(k_x\) and \(k_y\) directions (assuming the grid sizes are the same in both directions), and the number of sweeping grid points \(N\) for \(m/t_2\) and \(\\phi\).

Dependencies

import numpy as np
import cmath
from math import pi, sin, cos, sqrt

First subproblem

Subproblem description

Write a Haldane model Hamiltonian on a hexagonal lattice, given the following parameters: wavevector components \(k_x\) and \(k_y\) (momentum) in the x and y directions, lattice spacing \(a\), nearest-neighbor coupling constant \(t_1\), next-nearest-neighbor coupling constant \(t_2\), phase \(\\phi\) for the next-nearest-neighbor hopping, and the on-site energy \(m\).

Function header

def calc_hamiltonian(kx, ky, a, t1, t2, phi, m):
    '''Function to generate the Haldane Hamiltonian with a given set of parameters.
    Inputs:
    kx : float
        The x component of the wavevector.
    ky : float
        The y component of the wavevector.
    a : float
        The lattice spacing, i.e., the length of one side of the hexagon.
    t1 : float
        The nearest-neighbor coupling constant.
    t2 : float
        The next-nearest-neighbor coupling constant.
    phi : float
        The phase ranging from -\u03c0 to \u03c0.
    m : float\n        The on-site energy.
    Output:
    hamiltonian : matrix of shape(2, 2)
        The Haldane Hamiltonian on a hexagonal lattice.
    '''

Return line

    return hamiltonian

Example test case

kx = 1
ky = 1
a = 1
t1 = 1
t2 = 0.3
phi = 1
m = 1
assert np.allclose(calc_hamiltonian(kx, ky, a, t1, t2, phi, m), target)"

Second subproblem

Subproblem description

Calculate the Chern number using the Haldane Hamiltonian, given the grid size \(\\delta\) for discretizing the Brillouin zone in the \(k_x\) and \(k_y\) directions (assuming the grid sizes are the same in both directions), the lattice spacing \(a\), the nearest-neighbor coupling constant \(t_1\), the next-nearest-neighbor coupling constant \(t_2\), the phase \(\\phi\) for the next-nearest-neighbor hopping, and the on-site energy \(m\).”

Function header

def compute_chern_number(delta, a, t1, t2, phi, m):
    '''Function to compute the Chern number with a given set of parameters.
    Inputs:
    delta : float
        The grid size in kx and ky axis for discretizing the Brillouin zone.
    a : float
        The lattice spacing, i.e., the length of one side of the hexagon.
    t1 : float
        The nearest-neighbor coupling constant.
    t2 : float
        The next-nearest-neighbor coupling constant.
    phi : float
        The phase ranging from -\u03c0 to \u03c0.
    m : float
        The on-site energy.
    Output:
    chern_number : float
        The Chern number, a real number that should be close to an integer. The imaginary part is cropped out due to the negligible magnitude.
    '''#### Return line

Return line

    return chern_number

Example test case

delta = 2 * np.pi / 30
a = 1.0
t1 = 4.0
t2 = 1.0
N = 40
assert cmp_tuple_or_list(compute_chern_number_grid(delta, a, t1, t2, N), target)

Third subproblem

Subproblem description

Make a 2D array of Chern numbers by sweeping the parameters: the on-site energy to next-nearest-neighbor coupling ratio (\(m/t_2\) from -6 to 6 with \(N\) samples) and phase (\(\\phi\) from \(-\\pi\) to \(\\pi\) with \(N\) samples) values. Given the grid size \(\\delta\) for discretizing the Brillouin zone in the \(k_x\) and \(k_y\) directions (assuming the grid sizes are the same in both directions), the lattice spacing \(a\), the nearest-neighbor coupling constant \(t_1\), and the next-nearest-neighbor coupling constant \(t_2\).

Function header

def compute_chern_number_grid(delta, a, t1, t2, N):
    '''Function to calculate the Chern numbers by sweeping the given set of parameters and returns the results along with the corresponding swept next-nearest-neighbor coupling constant and phase.
    Inputs:
    delta : float
        The grid size in kx and ky axis for discretizing the Brillouin zone.
    a : float
        The lattice spacing, i.e., the length of one side of the hexagon.
    t1 : float
        The nearest-neighbor coupling constant.
    t2 : float
        The next-nearest-neighbor coupling constant.
    N : int
        The number of sweeping grid points for both the on-site energy to next-nearest-neighbor coupling constant ratio and phase.
    Outputs:
    results: matrix of shape(N, N)
        The Chern numbers by sweeping the on-site energy to next-nearest-neighbor coupling constant ratio (m/t2) and phase (phi).
    m_values: array of length N
        The swept on-site energy to next-nearest-neighbor coupling constant ratios.
    phi_values: array of length N
        The swept phase values.
    '''

Return line

    return results, m_values, phi_values

Example test case

delta = 2 * np.pi / 30
a = 1.0
t1 = 4.0
t2 = 1.0
N = 40
assert cmp_tuple_or_list(compute_chern_number_grid(delta, a, t1, t2, N), target)

Changelog

[3-A] - 2026-08-27

• Fix extract_code to take the content of markdown code fences instead of only deleting the fence markers, which left any prose outside the fences in the code that was run. Because the scorer runs each subproblem’s code again in every later subproblem of the same problem, one such completion also failed all later subproblems. All four metrics (percentage_main_problems_solved, percentage_subproblems_solved, total_main_problems_solved, total_subproblems_solved) can change, in both directions. Up for completions with prose before a closed fence or after it (in archived gpt-4o logs, prose after the closing fence affected 14 of 301 completions and 43 downstream subproblems), for fences labelled Python, python3 or py, and for an unclosed fence after prose. Down only for a completion with several fenced blocks where code outside the chosen block is needed and the fence-stripped text also parses, e.g. a constant defined in another block. Of several blocks, the last one that defines the subproblem’s function is used (so a corrected solution wins over an earlier draft, and a trailing usage example or fragment is skipped), otherwise the first block. When the fence-stripped text parses and defines a function or class the chosen block lacks, the fence-stripped text is used, so a solution split across blocks is unaffected. An unclosed fence (a truncated completion) and a completion with no fence give the same code as before. Replayed on 11,417 GLM-5.3 SciCode completions, 273 that failed to parse now parse and none that parsed now fails or loses a definition. Matches the original SciCode harness’s first python block except when a later block also defines the function, or no block does; see Evaluation. (@tuandin-subq)

[2-A] - 2026-02-16

• Migrate version to new scheme. See #907.

Static checks

Results of inspect-evals-lint at the commit this documentation was built from. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…