Contribution Guide

We welcome new evaluations and improvements to existing evaluations. You can contribute by installing the development version of inspect_evals, adding your evaluation, and opening a pull request in this repo.

Development

To develop a new evaluation:

Clone the repository and install the package with the -e flag and [dev] optional dependencies:

git clone https://github.com/UKGovernmentBEIS/inspect_evals.git
cd inspect_evals
pip install -e ".[dev]"

Create a sub-directory in src/inspect_evals for your evaluation and add your evaluation task and related code.
Create an __init__.py file in the directory and use it to export your task and other useful functions.
Register your task(s) for execution by the inspect eval CLI command by importing into src/inspect_evals/_registry.py.
Confirm that your evaluation is properly registered by running it:
```
inspect eval inspect_evals/<my-evaluation> 
```

Submission

To prepare an evaluation for submission as a pull request:

Create a README.md file in your sub-directory following the template used by other evaluations.
Run linting, formatting, and type checking and address any issues:
```
make check
```
Note that we automatically run the make check commands on all incoming PRs, so be sure your contribution passes those before opening a PR.
If an evaluation is based on an existing paper or ported from another evaluation suite, you should validate that it roughly matches the results reported there before submitting your PR. This is to ensure that your new implementation faithfully reproduces the original evaluation.

Comparison results between existing implementations and your implementation should be included in the pull request description.

Examples

Here are some existing evaluations that serve as good examples of what is required in a new submission:

GPQA, a simple multiple-choice evaluation
GSM8K, a mathematics task with fewshot prompting,
HumanEval, a Python coding task.
InterCode, a capture the flag (CTF) cybersecurity task.