Humanity’s Last Exam

Humanity’s Last Exam is a QA benchmark containing 2,500 questions across 100+ subjects, including mathematics, sciences, and humanities. About a quarter of the questions are multiple-choice and the rest open-ended, and about 14% include images. The prompt requests reasoning, a final answer, and 0-100% confidence; LLM judges mark correctness to produce accuracy and calibration metrics.

Contributed By

@SasankYadati

Code

src/inspect_evals/hle

Paper

https://arxiv.org/abs/2501.14249

Overview

Humanity’s Last Exam is a QA benchmark containing 2,500 questions across 100+ subjects, including mathematics, sciences, and humanities. The prompt requests reasoning, a final answer, and 0-100% confidence; LLM judges mark correctness to produce accuracy and calibration metrics.

HLE Verified Gold is a subset of 668 questions highlighted by Zhai et al (2026) as being higher quality. We use it as the default because it is likely to minimise both label noise and cost.

Configuration options

default.yaml sets the recommended defaults for Inspect’s HLE implementation

original.yaml provides an (almost) exact replication of the implementation used by CAIS

Task Arguments

Task argument	Options and default	Description
`answer_type`	`null` (default), `exactMatch`, `multipleChoice`	Filter for open-ended or MCQ questions
`category`	`null` (default); to filter by `category`, use a string or list of strings	Categories are `Biology/Medicine`, `Chemistry`, `Computer Science/AI`, `Engineering`, `Humanities/Social Science`, `Math`, `Other`, and `Physics`. This filter combines with `subject` and the other filters.
`subject`	`null` (default); to filter by `subject`, use a string or list of strings	Examples of subjects include `Genetics`, `Quantum Physics`, and `Organic Chemistry`. This filter combines with `category` and the other filters.
`include_multi_modal`	`true` (default), `false`	Includes questions with images when `true`. Set `false` for a text-only questions
`only_hle_verified_gold`	`true` (default), `false`	Filters for the 668 “gold” items identified by authors of the HLE-Verified paper. Combined with `rolling=true`, it keeps the intersection (543 items)
`rolling`	`false` (default), `true`	Chooses the static 2,500-question `cais/hle` dataset when `false`, or the pinned 2,689-question `cais/hle-rolling` snapshot when `true`.
`system_prompt`	`system_mc` (default), `system_exact_answer`	Selects the system prompt added to every sample. `system_mc` is the CAIS leaderboard prompt for all question types; `system_exact_answer` is the paper v1 exact-answer format.
`scorer`	A scorer spec dict; the built-in judges are `inspect_evals/hle/regex_judge` (default) and `inspect_evals/hle/json_judge`	Controls the judge prompt, parsing, model roles, and retries. See configs for all options.

Set these arguments with -T name=value on the command line or under task.args in a run config. Dataset filters combine, so report every filter used. For the CAIS judge setup, use run_configs/original.yaml.

System Prompt and Solver

Set system_prompt under task.args in the run config, or use -T system_prompt=system_mc on the command line. The task resolves the prompt from prompts.py and includes it as a system message in every dataset sample. Built-in scorer choices: graders selects model roles (default: [grader, grader_2]), each producing a separate score column and adding judging cost. args selects prompt titles and max_grader_attempts (default: 3; 1 matches the CAIS single pass). The sibling metrics field selects metrics and their arguments for each judge. Bind judge models through --model-role or the run config’s model_roles.

task:
  task: inspect_evals/hle
  args:
    system_prompt: system_mc

solver:
  solver: generate
  args:
    tool_calls: "none"

system_mc is the CAIS leaderboard prompt used for every answer type. system_exact_answer is the paper v1 exact-answer format, which the CAIS pipeline never sent. Both shipped configs use system_mc and Inspect’s generate solver with tool calls disabled.

Metrics

Metric	What it answers	Important caveat
`hle/accuracy`, `hle/stderr`	Accuracy over samples with parseable judge results.	This is Inspect’s standard scored-only convention, so unscored samples are excluded; NaN when nothing was scored. Every HLE metric and scorer is registered under an eval-qualified `hle/` name; bare `accuracy` and `stderr` would also collide with other registrations.
`hle/original_accuracy`	CAIS accuracy as a percentage, rounded to two decimal places.	Uses the selected static or rolling dataset size after dataset filters and before run limits. Missing judgments and omitted questions stay in the denominator. Requires one attempt per question.
`hle/cerr`	RMS calibration error over confidence bins on a 0-1 scale.	Not comparable with CAIS leaderboard Calibration Error: the CAIS implementation drops the top-confidence bin and reports integer percent.
`hle/unscored`	Number of samples whose judgment could not be parsed.	These samples are excluded from `hle/accuracy`, `hle/stderr`, and `hle/cerr`.

Requirements

Accept the access conditions for cais/hle on Hugging Face, and set HF_TOKEN to a token from that account with read access to the dataset. With rolling=true, also accept the conditions for cais/hle-rolling, which is separately gated.
Configure credentials for the model being evaluated and the judges. The default judges require OPENROUTER_API_KEY; the CAIS judge in run_configs/original.yaml requires OPENAI_API_KEY.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/hle --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.hle import hle
eval(hle)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/hle --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/hle --max-connections 10
uv run inspect eval inspect_evals/hle --temperature 0.5

See uv run inspect eval --help for all available options.

Dataset Details

Questions carry an answer_type of exactMatch (about 76%) or multipleChoice (about 24%). Every question is sent with the same system prompt, matching the CAIS methodology described in Changelog [5-C].

System: Your response should be in the following format:
Explanation: {your explanation for your answer choice}
Answer: {your chosen answer}
Confidence: {your confidence score between 0% and 100% for your answer}

User: For this question we work in ZF set theory without the axiom of choice. For a positive integer n, let AC(n) denote the sentence "every family of n-element sets has a nonempty product". What is the largest positive integer n such that AC(2) implies AC(n)?

About 14% of questions include an image alongside the text (342 of 2,500 at the pinned revision). Running with -T include_multi_modal=false -T only_hle_verified_gold=false evaluates the 2,158-question text-only subset, which corresponds to the CAIS leaderboard’s separate “Text-Only” convention; with the gold-subset default left on, the flag alone yields the text-only gold questions.

HLE-Verified Gold Subset

Independent audits have found a substantial error rate in HLE’s reference answers. HLE-Verified (dataset, code) reviews all 2,500 items and sorts them into a gold subset (668 items judged sound and left unmodified), a revision subset (1,143 items whose answers it corrects), and an uncertain subset (689 items it could not resolve).

This implementation reads only the gold item ids from HLE-Verified, pinned to revision 0bc8364; the questions and answers still come from the pinned cais/hle revision, so no revised answers are used. Scores are not comparable with the CAIS leaderboard, and the gold subset is not a random sample of full HLE.

HLE-Verified is a third-party re-annotation, not an CAIS fork or artifact of the HLE authors, and its own judgements are themselves LLM-assisted. The paper’s abstract describes a 641-item gold subset; the published dataset used here contains 668.

Rolling Dataset

-T rolling=true evaluates a pinned snapshot of cais/hle-rolling instead of the static set. This is the authors’ dynamic fork of the original HLE dataset, continually updated with feedback from the research community and intended to replace easier questions with harder held-out ones as frontier models improve. It holds 2,689 questions at the pinned revision 9437b15.

Rolling is pinned like the static set: “rolling” means upstream moves, not that a run should change underneath you. Advancing the pin changes which questions are asked, so treat it as a task change. Always report the revision alongside the score.

rolling can be combined with only_hle_verified_gold, and the result is the intersection of the two: the gold questions the rolling set retains (543 of the 668 gold ids at the pinned revisions). Note that the audit covers only the static set, so none of the 451 questions rolling added carry a gold label and all of them are excluded by the filter.

Judging Details

Two run configs are provided in Inspect’s single-file --run-config format. run_configs/original.yaml is paper-faithful: the CAIS prompt, full static dataset, one judging pass, and openai/o3-mini-2025-01-31 as judge. run_configs/default.yaml evaluates the HLE-Verified gold subset instead and grades with this port’s pre-5-C regex_judge prompt and validated lower-cost OpenRouter judges.

The maintained default runs two independent OpenRouter judges: the primary, lower-cost fp8 openrouter/z-ai/glm-5.3-flash, and openrouter/google/gemini-3.7-flash as a backup judge close to the frontier. In the validation used for this port, both tracked the consensus more closely than the CAIS openai/o3-mini-2025-01-31 judge. For a single headline number, use the primary judge; the backup adds redundancy, and disagreement between the two flags noisy grading. The default run emits both as separate columns rather than combining them.

How the judges were chosen: candidate judges graded Opus 5 and GPT-5.6 answers to HLE questions via OpenRouter, and each verdict was scored against a pseudo-ground-truth from a three-family frontier panel (Opus 5, GPT-5.6 and Gemini 3.1 Preview); 112 of the 4,943 panel decisions were contested and were settled by a deliberation round, then majority vote. glm-5.3-flash was the Pareto-frontier judge among candidates both cheaper and more accurate than the CAIS o3-mini judge; gemini-3.7-flash costs about as much as o3-mini but agreed with the panel more closely, and is kept as the backup. OpenRouter list prices on 2026-09-07: glm-5.3-flash $0.15/M input and $0.50/M output at most fp8 endpoints, gemini-3.7-flash $0.375/M and $1.875/M, o3-mini $1.10/M and $4.40/M. Per-judge agreement rates, the full candidate list and the deliberation code are in that post.

default.yaml is also the source of the eval’s built-in defaults: its task.args supply hle()’s parameter defaults, and its model_roles.grader is the judge used when no grader role is bound. A plain uv run inspect eval inspect_evals/hle therefore runs the same way as passing the file explicitly, and editing the file changes the task. Command-line flags such as --temperature, -T, and --model-role still override it.

Model responses are graded by default by two independent LLM judges, the grader and grader_2 model roles. They produce separate score columns (hle/regex_judge and hle/regex_judge1; hle/json_judge under original.yaml) with their own metrics; the judges are not voted or averaged against each other. Use original.yaml, or a scorer spec whose graders list names one role, for a single-judge run.

Each judge role resolves through --model-role or a run config, falling back to that role’s stanza in run_configs/default.yaml. With no override, the primary judge is openrouter/z-ai/glm-5.3-flash and grader_2 is openrouter/google/gemini-3.7-flash, so default judging requires OPENROUTER_API_KEY. A bound role inherits any generation settings it leaves unset (for example max_tokens and temperature) from its stanza, but it replaces the model itself and does not inherit model_args such as the default GLM fp8 provider filter:

uv run inspect eval inspect_evals/hle --model <provider/model> \
    --model-role grader=openai/o3-mini-2025-01-31

For an OpenRouter grader override with model-specific routing settings, put the provider mapping inside the role’s model_args:

uv run inspect eval inspect_evals/hle --model <provider/model> \
    --model-role 'grader={model: openrouter/qwen/qwen3.6-27b, model_args: {provider: {quantizations: [fp8]}}, temperature: 0, max_tokens: 32768}'

For repeated runs, fork run_configs/default.yaml, change the model_roles.grader stanza there, and run the fork with --run-config. That keeps model-specific routing settings easier to review than a long inline --model-role value.

Neither run config sets a solver temperature. The CAIS README recommends 0, but the CAIS script never sends one, and reasoning models reject the parameter outright. Pass --temperature 0 to apply the recommendation on sampling models. CAIS guidance also says not to set the completion budget below 8192 tokens for reasoning models, because smaller budgets can leave no visible answer after reasoning-token spend.

Judge Scorers

The scorer task parameter selects and parameterizes the judge. It is a spec dict of the form {"scorer": <name>, "graders": [...], "args": {...}, "metrics": [...]}: the shared inspect_evals.utils.scorers_from_spec helper resolves names through Inspect’s registry, so inspect_evals/hle/regex_judge and inspect_evals/hle/json_judge select the judges in scorers.py, a bare name selects an inspect_ai built-in, and any other registered package/name (e.g. inspect_ai/model_graded_qa) also works. Only registered scorers resolve — the spec arrives as data, so an arbitrary importable callable named in it cannot be imported and called. A scorer supplied this way must emit the {"score", "confidence"} value shape for the HLE metrics to apply. A top-level graders list builds one scorer per named model role (the scorer must accept a grader_role argument), and prompt arguments in args are Prompt titles resolved against prompts.py.

inspect_evals/hle/json_judge (with the json_judge prompt title; original.yaml’s judge): the CAIS judge prompt, verbatim, requiring structured output (JSON schema) from the grader. The judge extracts the response’s final answer and self-reported confidence, exactly as in the CAIS pipeline.
inspect_evals/hle/regex_judge (default, with the regex_judge_template and regex_judge_instructions prompt titles): this port’s pre-5-C paraphrased prompt via model_graded_qa, asking the judge to end with GRADE: C or GRADE: I, with the last stated confidence extracted by regex from the candidate completion. It omits the CAIS prompt’s answer-extraction step and numerical-tolerance clause, and needs no JSON-schema support from the grader.

For example, the CAIS prompt with the default judges:

uv run inspect eval inspect_evals/hle --model <provider/model> \
    -T scorer='{scorer: inspect_evals/hle/json_judge, graders: [grader, grader_2], args: {prompt: json_judge, max_grader_attempts: 3}, metrics: [inspect_evals/hle/accuracy, inspect_evals/hle/stderr, inspect_evals/hle/cerr, inspect_evals/hle/unscored]}'

On both paths, an unusable judge response is retried before the sample is left unscored, up to the spec’s max_grader_attempts. default.yaml sets 3 because retrying recovers samples lost to transient judge failures; original.yaml sets 1 because the CAIS pipeline judges each response exactly once.

CAIS Judge Prompt Reference

The full prompt is included for reproducibility. Most runs only need the summary above: the json_judge scorer used by original.yaml uses structured output and judge-side answer/confidence extraction.

Judge whether the following [response] to [question] is correct or not based on the precise and unambiguous [correct_answer] below.

[question]: {question}

[response]: {response}

Your judgement must be in the format and criteria specified below:

extracted_final_answer: The final exact answer extracted from the [response]. Put the extracted answer as 'None' if there is no exact, final answer to extract from the response.

[correct_answer]: {correct_answer}

reasoning: Explain why the extracted_final_answer is correct or incorrect based on [correct_answer], focusing only on if there are meaningful differences between [correct_answer] and the extracted_final_answer. Do not comment on any background to the problem, do not attempt to solve the problem, do not argue for any answer different than [correct_answer], focus only on whether the answers match.

correct: Answer 'yes' if extracted_final_answer matches the [correct_answer] given above, or is within a small margin of error for numerical problems. Answer 'no' otherwise, i.e. if there if there is any inconsistency, ambiguity, non-equivalency, or if the extracted answer is incorrect.


confidence: The extracted confidence score between 0|\%| and 100|\%| from [response]. Put 100 if there is no confidence score available.

The |\%| artifacts and “if there if there” typo are present in the CAIS implementation’s prompt at runtime and are reproduced deliberately. This is a user message, and the judge sees text only, matching the CAIS text-only judge, so the grader does not need vision. The json_judge path requires a provider that supports JSON-schema structured output; strict schema adherence is available on OpenAI and Mistral.

Metric Details

hle/original_accuracy follows CAIS’s dump_metrics calculation: round(100 * correct / total_questions, 2). The loader records the filtered dataset’s size in each sample’s cais_dataset_size metadata. Static and rolling datasets therefore supply their own denominator automatically. Unlike accuracy, this metric remains relative to the filtered dataset when a run limit or missing judgment reduces the scored set. Questions excluded by dataset filters do not enter the denominator. No denominator or percentage option is needed in YAML.

This metric requires single-attempt results and the dataset-size metadata. It raises an error for missing or inconsistent metadata or multiple recorded attempts per question. With no sample scores, Inspect reports an undefined metric rather than inferring a dataset size. The remaining metrics retain their existing definitions; stderr and cerr do not reproduce CAIS’s confidence interval and calibration calculation.

The scorer spec’s metrics field selects registered @metric factories for that scorer. Set the selection, order, and factory arguments alongside scorer, graders, and args, for example:

task:
  task: inspect_evals/hle
  args:
    scorer:
      scorer: inspect_evals/hle/regex_judge
      graders: [grader, grader_2]
      args:
        template: regex_judge_template
        instructions: regex_judge_instructions
        max_grader_attempts: 3
      metrics:
        - inspect_evals/hle/accuracy
        - inspect_evals/hle/cerr
        - confidence:
            - metric: inspect_ai/mean
              args: {}

Names are Inspect registry names: everything HLE defines in metrics.py and scorers.py is inspect_evals/hle/<name>, shown in logs and the viewer as hle/<name>; a bare name is an inspect_ai built-in, and package/name selects any other registered metric. No separate list of available metrics is maintained. HLE’s metrics consume the full {"score", "confidence"} value. Metrics from other packages must accept the selected scorer’s output shape; the example applies a scalar metric to the confidence field.

Omit the scorer spec’s metrics field or set it to null to retain that scorer’s declared metrics, or metrics: [] to disable aggregation. HLE’s own judge factories have no declared metrics, so null also disables their aggregation. The selection applies to each judge created from that spec without changing other scorer instances. Replacing the scorer spec replaces its metric selection too.

Judge parse failures in the regex_judge path surface as unscored on inspect_ai >= 0.3.245; on earlier versions model_graded_qa scores them INCORRECT instead. The json_judge path handles its own parsing and behaves identically across versions. This package’s floor is inspect_ai >= 0.3.259 (see pyproject.toml), so the earlier-versions caveat is historical: every supported install already has the unscored behaviour.

hle/accuracy and hle/stderr are computed after Inspect reduces per-epoch scores with the mean reducer, but hle/cerr is computed at attempt level. A task-pinned reducer preserves the per-epoch (score, confidence) pairs, so epoch-averaging cannot hide miscalibration (#2096). Unscored attempts are excluded from calibration under the same convention as unscored samples.

For ordinary HLE runs, pass --epochs N without --epochs-reducer. Supplying an explicit reducer is allowed by Inspect, but it overrides HLE’s task-pinned reducer, so hle/cerr falls back to the reduced sample values. The CAIS methodology is single-attempt. Omit hle/original_accuracy from the scorer spec when using multiple epochs.

Parameters

`hle`

answer_type (Optional[Literal['exactMatch', 'multipleChoice']]): Question type to include: “multipleChoice” or “exactMatch” (the dataset’s own answer_type values). If None, both types are included. Combines with the other dataset filters. (default: None)
category (str | list[str] | None): Category or categories to include. If None, all categories are included. (default: None)
subject (str | list[str] | None): Raw subject or subjects to include. If None, all subjects are included. (default: None)
include_multi_modal (bool): Whether to include multi-modal samples in the dataset. (default: True)
only_hle_verified_gold (bool): Restrict the dataset to HLE-Verified’s gold subset — the 668 items a third-party audit checked and left unmodified. Questions and answers still come from the pinned cais/hle revision. Scores on this subset are NOT comparable with the CAIS leaderboard. Combined with rolling, keeps the intersection: the gold questions the rolling set retains. (default: True)
rolling (bool): Evaluate cais/hle-rolling, the authors’ dynamic fork that swaps easier questions for harder held-out ones, instead of the static 2,500-question set. Scores are NOT comparable with the static set or with runs pinned to a different rolling revision. (default: False)
system_prompt (str): Title of the model-under-test system prompt, resolved from SYSTEM_MESSAGES in prompts.py. “system_mc” (default) is the prompt the CAIS leaderboard sends to every question; “system_exact_answer” is the paper v1 exact-answer prompt the CAIS pipeline never sent. (default: 'system_mc')
scorer (dict[str, Any]): Judge spec of the form {“scorer”: name, “graders”: […], “args”: {…}, “metrics”: […]}. Names resolve through Inspect’s registry: this eval’s judges are inspect_evals/hle/regex_judge and inspect_evals/hle/json_judge, a bare name is an inspect_ai built-in, and any other registered package/name also works. A top-level graders list builds one score column per model role (each judge multiplies judging cost); args holds the scorer’s own arguments, including prompt titles from prompts.py and max_grader_attempts (how many times to ask the judge before leaving a sample unscored; 1 matches the CAIS single judging pass). metrics selects ordered or key-scoped registered metrics by the same naming rule (inspect_evals/hle/accuracy, inspect_evals/hle/cerr, …), with optional factory arguments. Omitted or None preserves scorer defaults; an empty list disables metrics. (default: {'scorer': 'inspect_evals/hle/regex_judge', 'metrics': ['inspect_evals/hle/accuracy', 'inspect_evals/hle/stderr', 'inspect_evals/hle/cerr', 'inspect_evals/hle/unscored'], 'graders': ['grader', 'grader_2'], 'args': {'template': 'regex_judge_template', 'instructions': 'regex_judge_instructions', 'max_grader_attempts': 3}})

Available Categories

The category parameter accepts strings that are used to filter the dataset. The dataset contains the following category values:

Biology/Medicine
Chemistry
Computer Science/AI
Engineering
Humanities/Social Science
Math
Other
Physics

Available Subjects

The subject parameter also accepts strings to filter the dataset by the subject column (e.g. Genetics, Quantum Physics, Organic Chemistry). The full list of the 193 subjects can be found in the HuggingFace dataset.

Changelog

[8-E] - 2026-10-13

An empty or whitespace-only model response is scored {"score": "I", "confidence": 100} with reason="no_response", and the judge is not called. Both regex_judge and json_judge do this. Previously the judge graded the empty answer, and a judge that failed to parse its reply left the sample unscored. Such samples now stay in the accuracy denominator as incorrect. The confidence of 100 matches what regex_judge already used when the response stated no confidence.

[7-E] - 2026-09-10

BREAKING (interface): the scorer spec is resolved by the shared inspect_evals.utils.scorers_from_spec, so scorer and metrics names are Inspect registry names, and everything HLE defines is registered under an eval-qualified name: inspect_evals/hle/regex_judge, inspect_evals/hle/json_judge, inspect_evals/hle/accuracy, inspect_evals/hle/stderr, inspect_evals/hle/original_accuracy, inspect_evals/hle/cerr and inspect_evals/hle/unscored. Bare names are inspect_ai built-ins and no longer resolve in scorers.py or metrics.py. Logs and the viewer show these names without the package prefix, so the score columns are now hle/regex_judge and hle/regex_judge1 (hle/json_judge under original.yaml) and the metric rows hle/accuracy, hle/stderr, hle/original_accuracy, hle/cerr and hle/unscored; update log analysis keyed on the old names. HLE trials this convention ahead of a repo-wide decision.
BREAKING (comparability): default.yaml, and therefore the task defaults, set only_hle_verified_gold: true, so a plain run evaluates the 668-item HLE-Verified gold subset instead of the full 2,500-question set. original.yaml keeps the full CAIS dataset. Pass -T only_hle_verified_gold=false to restore the previous default dataset.
original_accuracy now reports round(100 * correct / total_questions, 2) as CAIS does, replacing the previous 0-1 fraction over supplied samples. The denominator is the selected static or rolling dataset size after dataset filters and before run limits, recorded in sample metadata. Missing judgments and omitted questions stay in the denominator. The metric requires one attempt per question and dataset-size metadata. Other metric calculations are unchanged.

[6-E] - 2026-09-10

Add configurable metrics to the scorer spec. Each shipped config now selects its own metrics: default.yaml reports hle/accuracy, hle/stderr, hle/cerr and hle/unscored, and original.yaml reports hle/original_accuracy, hle/stderr, hle/cerr and hle/unscored. Previously every judge reported all five, so a default run no longer emits hle/original_accuracy (which requires one attempt per question) and the CAIS run no longer emits scored-only accuracy. Score calculations are unchanged. Metric factories are resolved by registration, including factory arguments and key-scoped selections. HLE judge factories no longer attach metrics when used directly in another task; supply that task’s metrics explicitly.

[6-D] - 2026-09-09

BREAKING (comparability): replace the default judges. The primary grader role moves from openrouter/google/gemma-4-31b-it to openrouter/z-ai/glm-5.3-flash, pinned to fp8 provider quantization so OpenRouter cannot route the judge to another precision, and grader_2 from openrouter/google/gemini-3.6-flash to openrouter/google/gemini-3.7-flash, both selected in a judge comparison against a frontier-panel consensus. Both judges’ max_tokens rises from 16384 to 32768 as headroom for the reasoning tokens these judges spend inside the completion budget. Default-run scores are not comparable with earlier versions; original.yaml and --model-role overrides are unaffected.
BREAKING (comparability): default.yaml grades with regex_judge — this port’s pre-5-C paraphrased prompt, GRADE: C/GRADE: I parsing and the candidate’s last stated confidence — instead of the CAIS prompt with structured output, so default-run scores stay comparable with this port’s pre-5-C runs. This reverses the 5-C default-prompt change for the maintained default only, and gives up the CAIS prompt’s answer-extraction step and numerical-tolerance clause. original.yaml keeps the CAIS json_judge.
BREAKING (interface): the accuracy and stderr metrics move onto each judge’s own results entry, listed first, so the log viewer’s headline is accuracy rather than cerr. They were previously score/accuracy and score/stderr on a separate score entry; update log analysis that reads that entry. Values are unchanged except when nothing was scored, where both are now NaN rather than 0.0.
BREAKING (interface): judge selection moved out of individual task parameters into a single scorer task argument, a spec dict resolved by the new scorers.py (a bare scorer name resolves there; a package/name resolves any registered scorer). The former judge_prompt, graders and max_grader_attempts parameters are now keys of the spec, and prompts are declared as Prompt objects with provenance in prompts.py and referenced by title. The judge score columns are renamed from llm_grader/llm_grader1 to the scorer names (regex_judge/regex_judge1 on the default run, json_judge under original.yaml); update any log-parsing that keys on the old names.
Grader roles now resolve through inspect’s get_model role machinery (requires inspect_ai >= 0.3.259). A bound judge inherits generation settings it leaves unset (e.g. max_tokens, temperature) from its default.yaml stanza — previously only max_tokens gap-filled, so a bound judge without an explicit temperature now judges at the stanza’s temperature 0. A grader role with no stanza raises inspect’s required-role error when unbound instead of an HLE-specific one.
-T rolling=true can now be combined with only_hle_verified_gold=true: the run keeps the intersection, the gold questions the rolling set retains. Previously the combination raised a ValueError.
Judge score metadata is now judge_model, judge_attempts and, for unscored samples, unscored_reason. The judge_prompt, judge_attempt_outputs, grader_failure_mode and grader_exception fields and the judge-resolution INFO log are removed; update log analysis that keys on them. regex_judge records the same keys and tags a grade-parse miss unscored_reason: grader_failed, the repo’s shared vocabulary, instead of model_graded_qa’s grade_parse_failure.
Add the answer_type task parameter to filter the dataset to multipleChoice or exactMatch questions (the dataset’s own answer_type values). Defaults to None (both types), so default behaviour is unchanged; combines with the other dataset filters (category, subject, only_hle_verified_gold).
Add the system_prompt task parameter, a title into the new SYSTEM_MESSAGES store in prompts.py (default system_mc, the prompt the CAIS leaderboard sends), so run configs state which system prompt every sample carries. The run configs also gain a solver section (inspect’s stock generate with tool_calls: "none"; the eval offers no tools), which hle() reads for its built-in solver. Default behaviour is unchanged.

[5-C] - 2026-08-19

Add a second default judge (openrouter/google/gemini-3.6-flash, the grader_2 role) as an independent, unreduced score column beside the primary judge. The graders task parameter selects which judge roles run; original.yaml keeps the single CAIS judge. Judging cost scales with the number of judges.
BREAKING (interface): removed the shuffle task parameter. Its fixed-seed permutation marked the dataset as shuffled, which made inspect’s native --sample-shuffle a silent no-op on default runs. The dataset now loads in natural order; use --sample-shuffle <seed> (with --limit N for a seeded random subset) instead of -T shuffle=true.
The grade_c_i path’s confidence regex now takes the response’s last stated confidence rather than the first, matching the CAIS judge-side extraction semantics (the prompt asks the model to end with its confidence).
Add -T only_hle_verified_gold=true to restrict the dataset to the 668-item gold subset of HLE-Verified, a third-party audit of all 2,500 HLE items. Questions and answers still come from the pinned cais/hle revision — only the item ids are read from the audit — so this is a filter, not a second dataset. Default behaviour is unchanged.
Add -T rolling=true to evaluate the authors’ rolling variant (cais/hle-rolling, pinned to 9437b15, 2,689 questions) instead of the static set. It cannot be combined with only_hle_verified_gold: only the static set was audited, so that filter would keep just the questions the two sets share and collapse the run to the static gold subset. Default behaviour is unchanged.
The judge retry count is now the max_grader_attempts task parameter instead of a hardcoded 3. run_configs/default.yaml sets 3; run_configs/original.yaml sets 1, matching the CAIS pipeline’s single judging pass.
BREAKING (comparability): adopt the unified CAIS system prompt for all questions. The previous dual exactMatch/multipleChoice scheme matched the HLE paper v1 text but not the CAIS leaderboard, whose exact-match branch was dead code; CAIS commit 67b3251 formalized the single prompt (paper v10 Appendix C.1.1). answer_type is now preserved in sample metadata.
BREAKING (comparability): the default judge prompt is now the CAIS judge prompt verbatim with structured output (judge_prompt=original); the previous paraphrased prompt remains available as judge_prompt=grade_c_i. On the original path the response’s confidence is extracted by the judge (CAIS semantics); confidence is clamped to [0, 100] on both paths.
The grader model role now defaults to the validated low-cost judge instead of self-grading: with no role bound, the grader is read from run_configs/default.yaml (openrouter/google/gemma-4-31b-it, requires OPENROUTER_API_KEY), so a plain run and that run config judge identically by construction. Override with --model-role grader=... or a run config.
Judge parse failures now propagate as unscored (after up to max_grader_attempts judge attempts) instead of silently corrupting metrics; GRADE: P no longer leaks partial credit into the binary grade; judge metadata (grading transcript, unscored reason) is preserved on scores. New original_accuracy (all-attempted denominator, CAIS convention) and unscored metrics make the accounting visible. cerr excludes unscored samples and returns NaN when nothing was scored.
Add run_configs/original.yaml (paper-faithful: the CAIS o3-mini judge and settings) and run_configs/default.yaml (same, with a validated low-cost judge). default.yaml doubles as the source of the eval’s built-in defaults — hle()’s parameter defaults and the unbound grader are read from it — so a plain run matches it and editing it changes the task.
Docs: dataset count corrected from 3,000 to 2,500 (the pinned revision); gated-access steps; text-only convention; metric conventions documented (cerr is not comparable with CAIS Calibration Error).

[4-B] - 2026-08-12

Compute calibration at attempt level when --epochs > 1: per-epoch attempts are preserved by a task-pinned reducer and cerr no longer measures the calibration of epoch-averaged attempts (see #2096).

[3-B] - 2026-04-01

Fix incorrect MIME types on image data URIs for known affected samples.

[2-B] - 2026-03-10

Add dataset subsetting support via the category and subject task parameters.

[2-A] - 2026-02-16

Migrate version to new scheme. See #907.

[1.1.0] - 2025-12-23

See Issue 709

Score structure updated for epoch compatibility

The scorer now returns a dictionary in Score.value containing score (CORRECT/INCORRECT) and confidence (0-100). Previously, confidence was stored in Score.metadata, which is not reduced across epochs. This change ensures the cerr (calibration error) metric works correctly when running evaluations with multiple epochs (--epochs).

Metric output structure change:

The metrics are now grouped by value key. The accuracy and stderr metrics are now under the score section.

Example output comparison:

# Before
llm_grader
accuracy    0.000
stderr      0.000
cerr        0.950

# After
llm_grader          score
cerr        0.700   mean    0.000
                    stderr  0.000

[1.0.1] - 2025-12-18

Adds backoff policy for functions that connect to huggingface servers.

Static checks

Results of inspect-evals-lint at the commit this documentation was built from. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…