Humanity’s Last Exam

Humanity’s Last Exam is a closed-ended academic benchmark of 2,500 expert-written questions across 100+ subjects, including mathematics, sciences, and humanities. About 14% of items include images, 24% are multiple-choice, and the remainder are exact-match. Items were selected to challenge frontier LLMs, then peer- and organiser-reviewed. The prompt requests reasoning, a final answer, and 0-100% confidence; LLM judges mark correctness to produce accuracy and calibration metrics.

Contributed By

@SasankYadati

Code

src/inspect_evals/hle

Paper

https://arxiv.org/abs/2501.14249

Overview

Humanity’s Last Exam is a closed-ended academic benchmark of 2,500 expert-written questions across 100+ subjects, including mathematics, sciences, and humanities. About 14% of items include images, 24% are multiple-choice, and the remainder are exact-match. Items were selected to challenge frontier LLMs, then peer- and organiser-reviewed. The prompt requests reasoning, a final answer, and 0-100% confidence; LLM judges mark correctness to produce accuracy and calibration metrics.

Researcher Guide

Start by choosing the research question. HLE has several legitimate run modes, but they answer different questions and their results are not directly comparable.

Choose The Run

Goal Use Main caveat
Compare with the official leaderboard uv run inspect eval --run-config src/inspect_evals/hle/run_configs/original.yaml --model <provider/model> Static 2,500-item dataset, multimodal included, one attempt, official openai/o3-mini-2025-01-31 judge.
Run the maintained default uv run inspect eval inspect_evals/hle --model <provider/model> Same task defaults as run_configs/default.yaml: static full dataset, this port’s regex_judge prompt (no answer-extraction step or numerical-tolerance clause, unlike the official prompt), primary fp8 openrouter/z-ai/glm-5.3-flash and backup openrouter/google/gemini-3.7-flash judges.
Evaluate a text-only model uv run inspect eval inspect_evals/hle --model <provider/model> -T include_multi_modal=false The 2,158-question text-only subset, excluding 342 questions with images.
Use the authors’ rolling question set uv run inspect eval inspect_evals/hle --model <provider/model> -T rolling=true Uses a pinned snapshot of cais/hle-rolling at revision 9437b15, not the static set. Scores are not comparable with the leaderboard, static HLE, or other rolling revisions.
Use the HLE-Verified gold subset uv run inspect eval inspect_evals/hle --model <provider/model> -T only_hle_verified_gold=true Uses the pinned HLE-Verified gold id list revision 0bc8364 to filter the static HLE questions. HLE-Verified is not an official fork, and results are not an unbiased estimate of full-HLE accuracy.

Report With Results

  • Dataset choice and revision: static cais/hle revision 5a81a4c, rolling cais/hle-rolling revision 9437b15, or HLE-Verified gold id list revision 0bc8364.
  • Whether multimodal questions were included, and any category or subject filters.
  • Judge setup: original.yaml, default.yaml, or explicit --model-role bindings; also report the scorer task argument if you changed it.
  • accuracy, original_accuracy, and unscored together when judge parse failures occur.
  • --epochs if greater than 1. For ordinary HLE runs, pass --epochs N without --epochs-reducer; an explicit reducer overrides the task-pinned attempt-level calibration path.

Read The Metrics

Metric What it answers Important caveat
accuracy, stderr Accuracy over samples with parseable judge results. This is Inspect’s standard scored-only convention, so unscored samples are excluded; NaN when nothing was scored.
original_accuracy Accuracy over all attempted samples, counting unparseable judgments as incorrect. This matches the official HLE accounting denominator and differs from accuracy whenever unscored > 0.
cerr RMS calibration error over confidence bins on a 0-1 scale. Not comparable with official leaderboard Calibration Error: the official implementation drops the top-confidence bin and reports integer percent.
unscored Number of samples whose judgment could not be parsed. These samples are excluded from accuracy, stderr, and cerr.

Before You Run

The default static dataset is gated: open the cais/hle dataset page while logged in to Hugging Face and accept the terms, then authenticate locally with huggingface-cli login or set HF_TOKEN. Rolling runs use cais/hle-rolling, which is separately gated.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/hle --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.hle import hle
eval(hle)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/hle --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/hle --max-connections 10
uv run inspect eval inspect_evals/hle --temperature 0.5

See uv run inspect eval --help for all available options.

Dataset Details

Questions carry an answer_type of exactMatch (about 76%) or multipleChoice (about 24%). Every question is sent with the same system prompt, matching the official methodology described in Changelog [5-C].

System: Your response should be in the following format:
Explanation: {your explanation for your answer choice}
Answer: {your chosen answer}
Confidence: {your confidence score between 0% and 100% for your answer}

User: For this question we work in ZF set theory without the axiom of choice. For a positive integer n, let AC(n) denote the sentence "every family of n-element sets has a nonempty product". What is the largest positive integer n such that AC(2) implies AC(n)?

About 14% of questions include an image alongside the text (342 of 2,500 at the pinned revision). Running with -T include_multi_modal=false evaluates the 2,158-question text-only subset, which corresponds to the official leaderboard’s separate “Text-Only” convention.

Rolling Dataset

-T rolling=true evaluates a pinned snapshot of cais/hle-rolling instead of the static set. This is the authors’ dynamic fork of the original HLE dataset, continually updated with feedback from the research community and intended to replace easier questions with harder held-out ones as frontier models improve. It holds 2,689 questions at the pinned revision 9437b15.

Rolling is pinned like the static set: “rolling” means upstream moves, not that a run should change underneath you. Advancing the pin changes which questions are asked, so treat it as a task change. Always report the revision alongside the score.

rolling can be combined with only_hle_verified_gold, and the result is the intersection of the two: the gold questions the rolling set retains (543 of the 668 gold ids at the pinned revisions). Note that the audit covers only the static set, so none of the 451 questions rolling added carry a gold label and all of them are excluded by the filter.

HLE-Verified Gold Subset

Independent audits have found a substantial error rate in HLE’s reference answers. HLE-Verified (dataset, code) reviews all 2,500 items and sorts them into a gold subset (668 items judged sound and left unmodified), a revision subset (1,143 items whose answers it corrects), and an uncertain subset (689 items it could not resolve).

This implementation reads only the gold item ids from HLE-Verified, pinned to revision 0bc8364; the questions and answers still come from the pinned cais/hle revision, so no revised answers are used. Scores are not comparable with the official leaderboard, and the gold subset is not a random sample of full HLE.

HLE-Verified is a third-party re-annotation, not an official fork or artifact of the HLE authors, and its own judgements are themselves LLM-assisted. The paper’s abstract describes a 641-item gold subset; the published dataset used here contains 668.

Judging Details

Two run configs are provided in Inspect’s single-file --run-config format. run_configs/original.yaml is paper-faithful: the official prompt, full static dataset, one judging pass, and openai/o3-mini-2025-01-31 as judge. run_configs/default.yaml keeps the same task defaults but grades with this port’s pre-5-C regex_judge prompt and validated lower-cost OpenRouter judges.

The maintained default runs two independent OpenRouter judges: the primary, lower-cost fp8 openrouter/z-ai/glm-5.3-flash, and openrouter/google/gemini-3.7-flash as a backup judge close to the frontier. In the validation used for this port, both tracked the consensus more closely than the official openai/o3-mini-2025-01-31 judge. For a single headline number, use the primary judge; the backup adds redundancy, and disagreement between the two flags noisy grading. The default run emits both as separate columns rather than combining them.

How the judges were chosen: candidate judges graded Opus 5 and GPT-5.6 answers to HLE questions via OpenRouter, and each verdict was scored against a pseudo-ground-truth from a three-family frontier panel (Opus 5, GPT-5.6 and Gemini 3.1 Preview); 112 of the 4,943 panel decisions were contested and were settled by a deliberation round, then majority vote. glm-5.3-flash was the Pareto-frontier judge among candidates both cheaper and more accurate than the official o3-mini judge; gemini-3.7-flash costs about as much as o3-mini but agreed with the panel more closely, and is kept as the backup. OpenRouter list prices on 2026-09-07: glm-5.3-flash $0.15/M input and $0.50/M output at most fp8 endpoints, gemini-3.7-flash $0.375/M and $1.875/M, o3-mini $1.10/M and $4.40/M. Per-judge agreement rates, the full candidate list and the deliberation code are in that post.

default.yaml is also the source of the eval’s built-in defaults: its task.args supply hle()’s parameter defaults, and its model_roles.grader is the judge used when no grader role is bound. A plain uv run inspect eval inspect_evals/hle therefore runs the same way as passing the file explicitly, and editing the file changes the task. Command-line flags such as --temperature, -T, and --model-role still override it.

Model responses are graded by default by two independent LLM judges, the grader and grader_2 model roles. They produce separate score columns (regex_judge and regex_judge1; json_judge under original.yaml) with their own metrics; the judges are not voted or averaged against each other. Use original.yaml, or a scorer spec whose graders list names one role, for a single-judge run.

Each judge role resolves through --model-role or a run config, falling back to that role’s stanza in run_configs/default.yaml. With no override, the primary judge is openrouter/z-ai/glm-5.3-flash and grader_2 is openrouter/google/gemini-3.7-flash, so default judging requires OPENROUTER_API_KEY. A bound role inherits any generation settings it leaves unset (for example max_tokens and temperature) from its stanza, but it replaces the model itself and does not inherit model_args such as the default GLM fp8 provider filter:

uv run inspect eval inspect_evals/hle --model <provider/model> \
    --model-role grader=openai/o3-mini-2025-01-31

For an OpenRouter grader override with model-specific routing settings, put the provider mapping inside the role’s model_args:

uv run inspect eval inspect_evals/hle --model <provider/model> \
    --model-role 'grader={model: openrouter/qwen/qwen3.6-27b, model_args: {provider: {quantizations: [fp8]}}, temperature: 0, max_tokens: 32768}'

For repeated runs, fork run_configs/default.yaml, change the model_roles.grader stanza there, and run the fork with --run-config. That keeps model-specific routing settings easier to review than a long inline --model-role value.

Neither run config sets a solver temperature. The official README recommends 0, but the official script never sends one, and reasoning models reject the parameter outright. Pass --temperature 0 to apply the recommendation on sampling models. Official guidance also says not to set the completion budget below 8192 tokens for reasoning models, because smaller budgets can leave no visible answer after reasoning-token spend.

Judge Scorers

The scorer task parameter selects and parameterizes the judge. It is a spec dict of the form {"scorer": <name>, "graders": [...], "args": {...}}: a bare name resolves to a scorer in scorers.py, and a package/name (e.g. inspect_ai/model_graded_qa) resolves any @scorer-registered factory. Only registered scorers resolve — the spec arrives as data, so an arbitrary importable callable named in it cannot be imported and called. A scorer supplied this way must emit the {"score", "confidence"} value shape for the HLE metrics to apply. A top-level graders list builds one scorer per named model role (the scorer must accept a grader_role argument), and prompt arguments in args are Prompt titles resolved against prompts.py.

  • json_judge (with the json_judge prompt title; original.yaml’s judge): the official judge prompt, verbatim, requiring structured output (JSON schema) from the grader. The judge extracts the response’s final answer and self-reported confidence, exactly as in the official pipeline.
  • regex_judge (default, with the regex_judge_template and regex_judge_instructions prompt titles): this port’s pre-5-C paraphrased prompt via model_graded_qa, asking the judge to end with GRADE: C or GRADE: I, with the last stated confidence extracted by regex from the candidate completion. It omits the official prompt’s answer-extraction step and numerical-tolerance clause, and needs no JSON-schema support from the grader.

For example, the official prompt with the default judges:

uv run inspect eval inspect_evals/hle --model <provider/model> \
    -T scorer='{scorer: json_judge, graders: [grader, grader_2], args: {prompt: json_judge, max_grader_attempts: 3}}'

On both paths, an unusable judge response is retried before the sample is left unscored, up to the spec’s max_grader_attempts. default.yaml sets 3 because retrying recovers samples lost to transient judge failures; original.yaml sets 1 because the official pipeline judges each response exactly once.

Official Judge Prompt Reference

The full prompt is included for reproducibility. Most runs only need the summary above: the json_judge scorer used by original.yaml uses structured output and judge-side answer/confidence extraction.

Judge whether the following [response] to [question] is correct or not based on the precise and unambiguous [correct_answer] below.

[question]: {question}

[response]: {response}

Your judgement must be in the format and criteria specified below:

extracted_final_answer: The final exact answer extracted from the [response]. Put the extracted answer as 'None' if there is no exact, final answer to extract from the response.

[correct_answer]: {correct_answer}

reasoning: Explain why the extracted_final_answer is correct or incorrect based on [correct_answer], focusing only on if there are meaningful differences between [correct_answer] and the extracted_final_answer. Do not comment on any background to the problem, do not attempt to solve the problem, do not argue for any answer different than [correct_answer], focus only on whether the answers match.

correct: Answer 'yes' if extracted_final_answer matches the [correct_answer] given above, or is within a small margin of error for numerical problems. Answer 'no' otherwise, i.e. if there if there is any inconsistency, ambiguity, non-equivalency, or if the extracted answer is incorrect.


confidence: The extracted confidence score between 0|\%| and 100|\%| from [response]. Put 100 if there is no confidence score available.

The |\%| artifacts and “if there if there” typo are present in the official implementation’s prompt at runtime and are reproduced deliberately. This is a user message, and the judge sees text only, matching the official text-only judge, so the grader does not need vision. The json_judge path requires a provider that supports JSON-schema structured output; strict schema adherence is available on OpenAI and Mistral.

Metric Details

Judge parse failures in the regex_judge path surface as unscored on inspect_ai >= 0.3.245; on earlier versions model_graded_qa scores them INCORRECT instead. The json_judge path handles its own parsing and behaves identically across versions. This package’s floor is inspect_ai >= 0.3.259 (see pyproject.toml), so the earlier-versions caveat is historical: every supported install already has the unscored behaviour.

accuracy and stderr are computed after Inspect reduces per-epoch scores with the mean reducer, but cerr is computed at attempt level. A task-pinned reducer preserves the per-epoch (score, confidence) pairs, so epoch-averaging cannot hide miscalibration (#2096). Unscored attempts are excluded from calibration under the same convention as unscored samples.

For ordinary HLE runs, pass --epochs N without --epochs-reducer. Supplying an explicit reducer is allowed by Inspect, but it overrides HLE’s task-pinned reducer, so cerr falls back to the reduced sample values. The official methodology is single-attempt; multi-epoch runs remain non-comparable with the leaderboard.

Parameters

hle

  • answer_type (Optional[Literal['exactMatch', 'multipleChoice']]): Question type to include: “multipleChoice” or “exactMatch” (the dataset’s own answer_type values). If None, both types are included. Combines with the other dataset filters. (default: None)
  • category (str | list[str] | None): Category or categories to include. If None, all categories are included. (default: None)
  • subject (str | list[str] | None): Raw subject or subjects to include. If None, all subjects are included. (default: None)
  • include_multi_modal (bool): Whether to include multi-modal samples in the dataset. (default: True)
  • only_hle_verified_gold (bool): Restrict the dataset to HLE-Verified’s gold subset — the 668 items a third-party audit checked and left unmodified. Questions and answers still come from the pinned cais/hle revision. Scores on this subset are NOT comparable with the CAIS leaderboard. Combined with rolling, keeps the intersection: the gold questions the rolling set retains. (default: False)
  • rolling (bool): Evaluate cais/hle-rolling, the authors’ dynamic fork that swaps easier questions for harder held-out ones, instead of the static 2,500-question set. Scores are NOT comparable with the static set or with runs pinned to a different rolling revision. (default: False)
  • system_prompt (str): Title of the model-under-test system prompt, resolved from SYSTEM_MESSAGES in prompts.py. “system_mc” (default) is the prompt the CAIS leaderboard sends to every question; “system_exact_answer” is the paper v1 exact-answer prompt the CAIS pipeline never sent. (default: 'system_mc')
  • scorer (dict[str, Any]): Judge spec of the form {“scorer”: name, “graders”: […], “args”: {…}}. A bare name resolves in scorers.py and a package/name resolves any registered scorer. A top-level graders list builds one score column per model role (each judge multiplies judging cost); args holds the scorer’s own arguments, including prompt titles from prompts.py and max_grader_attempts (how many times to ask the judge before leaving a sample unscored; 1 matches the CAIS single judging pass). (default: {'scorer': 'regex_judge', 'graders': ['grader', 'grader_2'], 'args': {'template': 'regex_judge_template', 'instructions': 'regex_judge_instructions', 'max_grader_attempts': 3}})

Available Categories

The category parameter accepts strings that are used to filter the dataset. The dataset contains the following category values:

  • Biology/Medicine
  • Chemistry
  • Computer Science/AI
  • Engineering
  • Humanities/Social Science
  • Math
  • Other
  • Physics

Available Subjects

The subject parameter also accepts strings to filter the dataset by the subject column (e.g. Genetics, Quantum Physics, Organic Chemistry). The full list of the 193 subjects can be found in the HuggingFace dataset.

Changelog

[6-D] - 2026-09-09

  • BREAKING (comparability): replace the default judges. The primary grader role moves from openrouter/google/gemma-4-31b-it to openrouter/z-ai/glm-5.3-flash, pinned to fp8 provider quantization so OpenRouter cannot route the judge to another precision, and grader_2 from openrouter/google/gemini-3.6-flash to openrouter/google/gemini-3.7-flash, both selected in a judge comparison against a frontier-panel consensus. Both judges’ max_tokens rises from 16384 to 32768 as headroom for the reasoning tokens these judges spend inside the completion budget. Default-run scores are not comparable with earlier versions; original.yaml and --model-role overrides are unaffected.
  • BREAKING (comparability): default.yaml grades with regex_judge — this port’s pre-5-C paraphrased prompt, GRADE: C/GRADE: I parsing and the candidate’s last stated confidence — instead of the official prompt with structured output, so default-run scores stay comparable with this port’s pre-5-C runs. This reverses the 5-C default-prompt change for the maintained default only, and gives up the official prompt’s answer-extraction step and numerical-tolerance clause. original.yaml keeps the official json_judge.
  • BREAKING (interface): the accuracy and stderr metrics move onto each judge’s own results entry, listed first, so the log viewer’s headline is accuracy rather than cerr. They were previously score/accuracy and score/stderr on a separate score entry; update log analysis that reads that entry. Values are unchanged except when nothing was scored, where both are now NaN rather than 0.0.
  • BREAKING (interface): judge selection moved out of individual task parameters into a single scorer task argument, a spec dict resolved by the new scorers.py (a bare scorer name resolves there; a package/name resolves any registered scorer). The former judge_prompt, graders and max_grader_attempts parameters are now keys of the spec, and prompts are declared as Prompt objects with provenance in prompts.py and referenced by title. The judge score columns are renamed from llm_grader/llm_grader1 to the scorer names (regex_judge/regex_judge1 on the default run, json_judge under original.yaml); update any log-parsing that keys on the old names.
  • Grader roles now resolve through inspect’s get_model role machinery (requires inspect_ai >= 0.3.259). A bound judge inherits generation settings it leaves unset (e.g. max_tokens, temperature) from its default.yaml stanza — previously only max_tokens gap-filled, so a bound judge without an explicit temperature now judges at the stanza’s temperature 0. A grader role with no stanza raises inspect’s required-role error when unbound instead of an HLE-specific one.
  • -T rolling=true can now be combined with only_hle_verified_gold=true: the run keeps the intersection, the gold questions the rolling set retains. Previously the combination raised a ValueError.
  • Judge score metadata is now judge_model, judge_attempts and, for unscored samples, unscored_reason. The judge_prompt, judge_attempt_outputs, grader_failure_mode and grader_exception fields and the judge-resolution INFO log are removed; update log analysis that keys on them. regex_judge records the same keys and tags a grade-parse miss unscored_reason: grader_failed, the repo’s shared vocabulary, instead of model_graded_qa’s grade_parse_failure.
  • Add the answer_type task parameter to filter the dataset to multipleChoice or exactMatch questions (the dataset’s own answer_type values). Defaults to None (both types), so default behaviour is unchanged; combines with the other dataset filters (category, subject, only_hle_verified_gold).
  • Add the system_prompt task parameter, a title into the new SYSTEM_MESSAGES store in prompts.py (default system_mc, the prompt the official leaderboard sends), so run configs state which system prompt every sample carries. The run configs also gain a solver section (inspect’s stock generate with tool_calls: "none"; the eval offers no tools), which hle() reads for its built-in solver. Default behaviour is unchanged.

[5-C] - 2026-08-19

  • Add a second default judge (openrouter/google/gemini-3.6-flash, the grader_2 role) as an independent, unreduced score column beside the primary judge. The graders task parameter selects which judge roles run; original.yaml keeps the single official judge. Judging cost scales with the number of judges.

  • BREAKING (interface): removed the shuffle task parameter. Its fixed-seed permutation marked the dataset as shuffled, which made inspect’s native --sample-shuffle a silent no-op on default runs. The dataset now loads in natural order; use --sample-shuffle <seed> (with --limit N for a seeded random subset) instead of -T shuffle=true.

  • The grade_c_i path’s confidence regex now takes the response’s last stated confidence rather than the first, matching the official judge-side extraction semantics (the prompt asks the model to end with its confidence).

  • Add -T only_hle_verified_gold=true to restrict the dataset to the 668-item gold subset of HLE-Verified, a third-party audit of all 2,500 HLE items. Questions and answers still come from the pinned cais/hle revision — only the item ids are read from the audit — so this is a filter, not a second dataset. Default behaviour is unchanged.

  • Add -T rolling=true to evaluate the authors’ rolling variant (cais/hle-rolling, pinned to 9437b15, 2,689 questions) instead of the static set. It cannot be combined with only_hle_verified_gold: only the static set was audited, so that filter would keep just the questions the two sets share and collapse the run to the static gold subset. Default behaviour is unchanged.

  • The judge retry count is now the max_grader_attempts task parameter instead of a hardcoded 3. run_configs/default.yaml sets 3; run_configs/original.yaml sets 1, matching the official pipeline’s single judging pass.

  • BREAKING (comparability): adopt the unified official system prompt for all questions. The previous dual exactMatch/multipleChoice scheme matched the HLE paper v1 text but not the official leaderboard, whose exact-match branch was dead code; official commit 67b3251 formalized the single prompt (paper v10 Appendix C.1.1). answer_type is now preserved in sample metadata.

  • BREAKING (comparability): the default judge prompt is now the official judge prompt verbatim with structured output (judge_prompt=original); the previous paraphrased prompt remains available as judge_prompt=grade_c_i. On the original path the response’s confidence is extracted by the judge (official semantics); confidence is clamped to [0, 100] on both paths.

  • The grader model role now defaults to the validated low-cost judge instead of self-grading: with no role bound, the grader is read from run_configs/default.yaml (openrouter/google/gemma-4-31b-it, requires OPENROUTER_API_KEY), so a plain run and that run config judge identically by construction. Override with --model-role grader=... or a run config.

  • Judge parse failures now propagate as unscored (after up to max_grader_attempts judge attempts) instead of silently corrupting metrics; GRADE: P no longer leaks partial credit into the binary grade; judge metadata (grading transcript, unscored reason) is preserved on scores. New original_accuracy (all-attempted denominator, official convention) and unscored metrics make the accounting visible. cerr excludes unscored samples and returns NaN when nothing was scored.

  • Add run_configs/original.yaml (paper-faithful: the official o3-mini judge and settings) and run_configs/default.yaml (same, with a validated low-cost judge). default.yaml doubles as the source of the eval’s built-in defaults — hle()’s parameter defaults and the unbound grader are read from it — so a plain run matches it and editing it changes the task.

  • Docs: dataset count corrected from 3,000 to 2,500 (the pinned revision); gated-access steps; text-only convention; metric conventions documented (cerr is not comparable with official Calibration Error).

[4-B] - 2026-08-12

  • Compute calibration at attempt level when --epochs > 1: per-epoch attempts are preserved by a task-pinned reducer and cerr no longer measures the calibration of epoch-averaged attempts (see #2096).

[3-B] - 2026-04-01

  • Fix incorrect MIME types on image data URIs for known affected samples.

[2-B] - 2026-03-10

  • Add dataset subsetting support via the category and subject task parameters.

[2-A] - 2026-02-16

  • Migrate version to new scheme. See #907.

[1.1.0] - 2025-12-23

See Issue 709

Score structure updated for epoch compatibility

The scorer now returns a dictionary in Score.value containing score (CORRECT/INCORRECT) and confidence (0-100). Previously, confidence was stored in Score.metadata, which is not reduced across epochs. This change ensures the cerr (calibration error) metric works correctly when running evaluations with multiple epochs (--epochs).

Metric output structure change:

The metrics are now grouped by value key. The accuracy and stderr metrics are now under the score section.

Example output comparison:

# Before
llm_grader
accuracy    0.000
stderr      0.000
cerr        0.950

# After
llm_grader          score
cerr        0.700   mean    0.000
                    stderr  0.000

[1.0.1] - 2025-12-18

  • Adds backoff policy for functions that connect to huggingface servers.