Evaluation Checklist

This checklist covers all the requirements you should need in order to produce a high-quality evaluation. It’s designed for full evaluations, and it’s too heavyweight for most PRs.

Usage of agents to help with coding and to complete agentic workflows is recommended but not mandatory:

1. Full list of agent workflows

2. Our philosophy on agents

3. Recommended permissions

Master Checklist

Contributors: ensure each step is done. Reviewers: verify each step has been done.

We recommend the use of agent workflows. Our agent workflows are optimised for use with Claude Code (setup instructions).

• Ensure the evaluation runs with ‘uv run inspect eval inspect_evals/ –limit 1’ on the model of your choice. (You can use more samples, but we recommend 5 or less at this early stage)
• Analyse the log, either with the Trajectory Analysis Workflow or manually. Fix it if it fails for spurious reasons.
• Run the Master Checklist Workflow which goes over several agent workflows in turn.
• Manually go over the Human-Required Checks section, and verify your evaluation meets each one.
• Check your implementation and results against the original paper or implementation, if one exists.
• Manually review your code, making sure it’s high-quality. All code must be reviewed, including LLM-generated code as per our LLM usage guidelines.
• Draft your PR, and name it something like <eval_name> implementation (putting the name of the evaluation first makes it easier to find your PR in a browser tab!). Push it to Github, and you’re ready for review!

Human-Required Checks

Best Practices

• Leverage Inspect components whenever possible
• Complex logic is commented
• Document and validate environment constraints

Evaluation Report

Evaluation Report Guidelines

We recommend using the Evaluation Report workflow to assist in this step. To verify the error rate of logs, use the Trajectory Analysis Workflow on the log files that emerge from the evaluation report.

• Logs are verified to allow end-to-end runs
• Logs are free of errors that crash the sample, and models are able to provide submissions.
• Logs have a 10% or lower rate of invalid samples
• All relevantly different subsets of the dataset pass here
• Results produced for at least two models, or reason why not clearly stated

Evaluation Report Notes

• Any changes that would cause deviations from the original evaluation are noted.
• Any limitations or edge cases of the evaluation are noted.

---

Agent Runnable Checks

The following items can be checked by an LLM agent with access to the codebase. These checks require reading code and comparing against conventions, but do not require running the evaluation or external context beyond the repository. If running the Review An Evaluation Workflow you don’t need to read these checks - they are here to serve as a reference in case of errors.

Code Quality (Agent)

• Existing naming conventions are followed
• Linting passes successfully (uv run ruff check will check this for you)
• Magic numbers in function defaults are extracted to named constants if they appear 3+ times or are not clear from context

Unit Tests (Agent)

• All custom solvers, scorers, datasets covered
• Custom tools are covered
• Custom utils/functions are covered
• Edge cases, error conditions, and invalid inputs are checked

End-to-End Tests (Agent)

• Each meaningfully different task/variant covered by E2E tests
• Tests are marked correctly with @mark items

Apply Pytest Marks (Agent)

• If a test triggers the download of a dataset, mark it with @pytest.mark.dataset_download, if it uses Huggingface also mark it with @pytest.mark.huggingface. Note that easily missed examples include E2E tests that instantiate a dataset, or solvers that pull a model from huggingface.
• If a test uses a Docker sandbox or otherwise triggers a docker build or pull, mark it with @pytest.mark.docker.

Best Practices - Task Design (Agent)

• Use model roles for multi-model evaluations, including model-graded ones
• Prompt templates defined as module-level constants, not inline
• Only call get_model() inside of solvers/scorers
• Separate prompt templates from formatting

Best Practices - Control Flow (Agent)

• Respect “no-limit” semantics
• Provide informative errors for invalid task parameters
• Provide defaults and allow overrides for datasets, solvers, scorers, metrics, and grader models

Best Practices - Datasets and Variants (Agent)

• Use stable, canonical IDs for samples
• Ensure deterministic behavior where possible
• Datasets are pinned to specific versions: all HuggingFace loading functions (hf_dataset, load_dataset, snapshot_download, hf_hub_download, sentence_transformer, transformers_pipeline) enforce revision= at runtime, and GitHub raw URLs use commit SHAs instead of branch names
• Differentiate tasks from dataset splits via parameters

Best Practices - Scoring and Metrics (Agent)

• Align scoring with the outcome

Best Practices - Documentation, Environments, and Tooling (Agent)

• Keep docs and defaults in sync
• Least-privilege tooling
• Keep dependency metadata and lockfiles in sync

Evaluation Report (Agent)

• Table is present containing results of evaluation run
• A comparison to the original paper is present, or its absence is justified
• Table contains the specific inspect eval command(s) used to produce it
• Full model names are mentioned explicitly (e.g, gpt-5.1-2025-11-13, not gpt-5.1)
• Evaluation version is mentioned explicitly
• Any inspect eval parameters used are justified within the report

This checklist is in beta

We will continue to improve this over time, and aim to make it our standard approach for both contributors and reviewers by the end of Q1 2026. Feedback on this checklist is very welcome - you may reach out via this Google Form, or open an issue/PR.