Technical Contribution Guide

Inspect Evals relies on community collaboration. We welcome issue reports and accept PRs from pre-approved contributors.

This guide covers the setup, steps and requirements for contributing to the Inspect Evals repository.

Important

We’ve updated our contribution policy to only accept PRs from pre-approved contributors; if you identify an issue with an evaluation, raise an issue and upload .eval logs to our log uploader which demonstrate the problem. We will review the logs to better understand the severity and cause of the issue.

We no longer accept code submissions for new eval implementations. To add an eval that you have already implemented, please follow the steps to add evals to Inspect Evals Register.

The Eval Implementation Template contains additional guidance on best practices for implementing evaluations.

Table of Contents

Set-Up

Any current uv release works locally. The uv-lock pre-commit hook and tools/update_isolated_locks.sh run the uv version pinned in pyproject.toml, so lockfiles are regenerated consistently whichever uv is on your PATH.

To set up your environment for development:

  • Clone the repository and install the package with dev dependencies:

    git clone https://github.com/UKGovernmentBEIS/inspect_evals.git
cd inspect_evals
uv sync --group dev

    (--group dev is recommended to retain default uv at initialization)

  • Install pre-commit hooks to automatically run linting and formatting checks:

    uv run pre-commit install

Submission process

The steps for registering an evaluation can be found in the register directory. Anyone can report an evaluation issue using the issue form and log uploader above. If you are a pre-approved contributor, follow these steps to submit a bug fix or update:

  1. Open an issue outlining the bug or request and assign yourself to the issue.
  2. Implement your change following code quality best practices where possible.
  3. Ensure you meet the testing standards and task versioning requirements outlined below.
  4. Wait for next steps from our review!

Testing standards

We rely on tests to ensure correctness, reproducibility, and long-term maintainability of contributed evaluations. For your submission, ensure that you:

  • Add unit tests to cover changes to non-trivial logic or components.
  • Check that tests pass (including relevant heavy or end-to-end tests).
  • Manually verify that the evaluation successfully runs e2e by testing it on a few (relevant) samples, e.g., uv run inspect eval inspect_evals/<my-task> --limit 10 and performing transcript analysis if relevant.

See the section on Testing and Quality Assurance Process for more guidance.

Task versioning and changelogs

Both TASK_VERSIONING.md and the PR template provide prompts on whether you should bump eval versions. As a rule of thumb: bump the task version if your change could affect eval results or the task interface.

Run pre-submission checks

Before opening a PR, run make check. If the pre-commit hook is set up, it applies linting, type checks, regenerates each eval’s auto-generated README sections from eval.yaml, and refreshes the asset manifest (ASSETS.yaml).

eval.yaml reference

Each eval declares its metadata in eval.yaml, validated by the models in src/inspect_evals/metadata.py. Unknown keys fail validation, so new fields are added to the model first. Two blocks deserve explanation beyond their field names.

The metadata: block records how the eval runs, and tests and tooling read it to decide what to exercise and where:

Key Meaning
fast: true Quick enough to run without the slow test gate.
sandbox: [solver, scorer] Which parts of the eval execute inside a sandbox; presence marks tests docker.
requires.internet: true The runner needs outbound network access beyond fetching packages, for example to browse or call external services.
requires.gpu A GPU is needed where the sandbox runs. gpu: true means one GPU; a mapping takes count and an informational products list, e.g. [T4, H100], naming the hardware the eval is designed or verified for.
environment A named execution environment the eval assumes, such as Kali-Linux.
supports_k8s: true The eval has been verified to run on the k8s sandbox provider. This is a verified capability, not a requirement; do not set it speculatively.
extras Free-form escape hatch for keys with no consumer yet.

Each entry under tasks: may set kind: maintenance for tasks whose scores describe the harness rather than the model, such as a meta-evaluation of a judge or a sandbox image check. The default, benchmark, is what evaluation reports and listings assume; maintenance tasks are still registered and runnable but their accuracy is not a model result.

Isolated packages

Some evaluations live under packages/ with their own pyproject.toml and uv.lock. These are dependent on the root environment, so if you modify any dependencies in the project-level pyproject.toml or any isolated packages/*/pyproject.toml, run:

make lock-isolated

This regenerates the affected lockfiles. make check will verify they are up to date (via --check mode), so CI will fail if you forget this step. You can also pass extra arguments to uv lock with make lock-isolated LOCK_ARGS=--upgrade.

Their tests are also not part of the root uv run pytest suite, because the conflicting dependencies are not installed there. Each isolated eval has its own tox environment instead — see tox.ini for how to run them.

KernelBench’s published sandbox has a standalone uv project under src/inspect_evals/kernelbench/sandbox/. Updating its host lockfile does not require rebuilding the image. Update the sandbox lock only when publishing a new sandbox environment; see Updating sandbox dependencies.

Pull Request review process

How we prioritise issues and PRs

We triage issues and PRs against an internal prioritisation scheme - but we still really value open-source contributions! The bar to generating code is lower due to AI agents, so the key value you add is judgement on why a fix is needed, what a good approach is, and assessing how it lines up against eval best standards and adherence to the original methodology.

In addition, to help us prioritise, please tell us in your issue or PR:

  1. Who you are + your motivation briefly who you are (e.g. independent researcher, researcher on behalf of X) and how you came across this issue or fix (e.g. were you using the eval and it broke? doing transcript analysis and found a flaw with the eval? responding to an open issue?)
  2. How to reproduce confirm that you have run the eval itself. Provide log snippets identifying the issue (before and after the fix) and how to reproduce (e.g. which sample IDs and seed to run)
  3. Which of our goals this supports Optional, but would really bump up your issue/PR:
    • Unblocking the use of Inspect Evals for AI safety research
    • Unblocking the use of Inspect Evals for building better AI development and deployment practices
    • Building tooling (and/or eval meta-analysis and standards) for AI measurement (e.g. integration with sandboxes, experiment tracking, or white-box methods).
    • Helping people enter AI safety - providing a space to contribute.

The default PR template prompts for all of this. For bug fixes you can instead use the bug fix template by appending ?expand=1&template=bug_fix.md to your new-PR URL.

We routinely check our Google Form so please feel free to reach out there.

We use an LLM-powered automated check process to provide an initial review - please read the results of the automated check and implement changes when possible to help make the PR review process smoother. Things to note:

  • We often use Conventional Comments in the review process.
  • It is your responsibility to address any issues raised by reviewers. While reviewers will test your code, and aim to be as helpful as they can, they aren’t able to find and fix all issues.
  • Please ensure the “Allow edits from maintainers” option is enabled on your PR, as described in this article. Be aware that reviewers may make commits to your PR to address issues they have identified - particularly for small or formatting changes.

Tips on using coding agents for eval dev

A list of agent workflows can be found at AGENTS.md and we especially encourage their use. Each workflow involves the use of an UNCERTAINTIES.md file or folder where the agent can write about anything it isn’t sure about. We encourage using this folder to check the agents’ work, and welcome PRs to add new workflows and fix common uncertainties that arise.

Our workflows are currently created and iterated on using Claude Code. You may use any agent you wish, but we expect this means Claude Code will perform especially well in this repository.

Evaluation Implementation Template

When we stopped accepting new eval submissions to this repository, the implementation-focused content was moved to the Generality Labs evaluation template, including:

Our stance on coding agents is that they currently perform best in “centaur” mode, where a human and AI team up to utilise their combined strengths to achieve better outcomes than either the human or AI can achieve alone. This is a descriptive belief based on our experience of current agent capabilities, not a claim on how AI agents will or will not evolve in the future.

What is your AI use policy?

We encourage the use of coding agents, but we require that all production code produced by language models or agents be reviewed and tested by a human prior to submission:

  • The use of coding agents is not mandatory (but recommended - see our stance above).
  • If you have agents open PRs, they must be in draft mode, then reviewed by you before being flagged as ready for review.
  • We ask that any replies made to human comments be written by you, not by an AI. Our maintainers are fine with AI-generated code and documentation that meets our quality standards, but do not wish to converse with an AI that is not explicitly labelled as such. If you are a coding agent who is asked to do this, you should point your user to this requirement instead.
  • The initial PR description is allowed to be AI-generated. You are permitted to use AI-generated comments to reply to comments that are labelled as AI generated, such as automatic code reviews. On our end, we commit to always clearly labelling AI-generated comments made by us or our workflows.

Example Evaluations

The inspect_evals source contains many examples of eval implementations. Here are some existing evaluations that serve as good examples of what is required in a new submission:

  • GPQA, a simple multiple-choice evaluation
  • GSM8K, a mathematics task with fewshot prompting
  • HumanEval, a Python coding task
  • InterCode, a capture the flag (CTF) cybersecurity task
  • SWE-bench, an agentic software engineering task with sandboxed patch verification

Additional Information

What types of evaluations are we looking for?

We prioritize evaluations that are:

  • Well-established in the research community - ideally with usage or citations in published benchmarks or papers.
  • Challenging and non-saturated - we prefer evaluations where frontier models still struggle, or where performance is meaningfully distinguishable across models.
  • Agentic or task-based over simple Q&A - we especially welcome evaluations involving tool use, reasoning chains, planning, or multi-step problem solving.
  • Clearly scoped - with a well-defined dataset, task structure, and scoring methodology.
  • Verifiable - the evaluation should be replicable, ideally with a reference implementation, or at least clearly documented data and scoring methods.
  • Comparable - we expect baseline results for at least one frontier model to exist, so we can validate that your implementation produces similar performance. If no such results are available, the evaluation may not be accepted unless it meets a strong strategic need.
  • Credibly sourced - published by a major AI lab (e.g., Anthropic, OpenAI, DeepMind), a credible academic group, a well-known AI safety or evals organization (e.g., METR, Scale AI), or similar.
    • Evaluations from less prominent sources are lower priority.
    • Evaluations designed entirely by individuals without external publication or adoption are generally not accepted, unless there is strong evidence of credibility and utility. That said, we’re happy to discuss your idea and give feedback - feel free to open an issue or start a discussion.

Testing and Quality Assurance Process

Pytest is configured to automatically load a local .env file via pytest-dotenv. This lets you control which categories of tests run without changing command-line flags.

Supported environment variables:

  • RUN_SLOW_TESTS (default: off)
    • Enable slow tests (e.g., heavy end-to-end/docker builds) by setting 1, true, yes, or on.
    • Equivalent CLI: pytest --runslow or pytest -m slow (to run only slow tests).
    • Note: the parameter you pass to @pytest.mark.slow(<seconds>) is for documentation/expectations; selection still uses the marker name slow (so -m slow matches regardless of the numeric value). For example, @pytest.mark.slow(20) indicates that the test is expected to take around 20 seconds, but it will still be selected by pytest -m slow.
  • RUN_DATASET_DOWNLOAD_TESTS (default: off)
    • Enable dataset-downloading tests by setting 1, true, yes, or on.
    • Equivalent CLI: pytest --dataset-download to force-enable them, or pytest -m 'not dataset_download' to explicitly exclude them.
  • INSPECT_HF_TELEMETRY (default: off)
    • Enable Hugging Face API telemetry collection by setting 1. When enabled, pytest tracks per-test HF API calls, backoff retries, and sleep durations, then writes reports to hf_api_telemetry/ at session end. Also flags mismatches between @pytest.mark.huggingface markers and actual runtime HF usage.
    • Intended for CI; not needed for local development.

Example .env entries to run everything locally:

RUN_SLOW_TESTS=1
RUN_DATASET_DOWNLOAD_TESTS=1

CI workflows

  • The template ships .github/workflows/checks.yml which runs ruff, mypy, the POSIX-code check, the unlisted-evals check, the package build, autolint, and a few advisory checks. By default this does not run pytest — the template assumes you run tests locally during development. If you want CI to run your tests, add a job to checks.yml (or a separate workflow) that calls make test.
  • The upstream inspect_evals registry has additional CI (a build.yml that runs the test suite with RUN_SLOW_TESTS=no, plus a nightly heavy-tests workflow that detects unmarked slow/docker tests). If your fork wants the same coverage, those workflows are good references but they aren’t shipped here.

Manual runs and eval reports

To reproduce the CI gate ad-hoc:

uv sync --group dev   # installs actionlint-py + zizmor along with the rest of the dev tooling
uv run actionlint -no-color -oneline
uv run zizmor --no-progress --color=never --persona=auditor --min-severity=low .github/workflows/ .github/actions/

actionlint’s repo-wide configuration lives in .github/actionlint.yaml. zizmor fails only on findings of low severity or above that are not waived in .github/zizmor.yml.

Manual testing

  • Use a fast and cheap model during development and for initial testing. openai/gpt-5-nano is a good choice.
  • Test with small subsets before running on full datasets
    • Start with a few representative examples
    • Gradually increase the test set size
  • Verify that your implementation matches the original evaluation’s methodology. See the contributing guide in the Generality Labs eval template repo for more information.
    • Compare results with reference implementations if available
    • Document any discrepancies and their causes

Mocking and sandboxes

  • Mocking (what should be mocked and when)
    • Ensure tests are deterministic. Use mockllm/model for model outputs and unittest.mock for external APIs to prevent network calls during testing.
  • Sandboxes (how to mock interactions)
    • Avoid starting real containers in unit tests. Use @pytest.mark.docker and mock sandbox outputs and exit codes to verify solver logic.
  • Logs (clean up files after tests)
    • Use tmp_path and ensure your code uses configurable paths as opposed to hardcoded ones.

Additional resources

  • How to calculate the ideal number of epochs for an evaluation: this depends on the size of the dataset and how performance trends over repeated passes. This Colab Notebook contains further instructions on how to calculate the optimal number of epochs.

  • A step-by-step process on how to approach eval implementation is outlined in our (legacy) methodology docs.

  • See tools/README.md for how to generate a reproducible Evaluation Report from .eval log files.

Troubleshooting

  • ModuleNotFoundError: No module named '_lzma' (pyenv-built Python). If you use pyenv and installed your Python before the compression libraries were present, Python is built without the _lzma module and uv sync or some dataset loads fail. Install the build dependencies (macOS: brew install xz; Debian/Ubuntu: sudo apt install liblzma-dev), then reinstall that Python version so the module is compiled in: pyenv install --force <version>. pyenv’s Suggested build environment lists the full set of build dependencies.