AIME 2024: Problems from the American Invitational Mathematics Examination

A benchmark for evaluating AI’s ability to solve challenging mathematics problems from the 2024 AIME - a prestigious high school mathematics competition.

Overview

The AIME 2024 benchmark is based on problems from the American Invitational Mathematics Examination, a prestigious high school mathematics competition.

This benchmark tests a model’s ability to solve challenging mathematics problems by generating step-by-step solutions and providing the correct final answer.

The benchmark uses the Maxwell-Jia/AIME_2024 dataset from Hugging Face.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/aime2024 --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.aime2024 import aime2024
eval(aime2024)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/aime2024 --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/aime2024 --max-connections 10
uv run inspect eval inspect_evals/aime2024 --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

aime2024

No task parameters.

Dataset

There are 30 problems in the dataset, and each problem includes:

• A mathematics problem from the AIME 2024
• The correct answer
• A step-by-step solution

Problems are structured:

{
  "ID": "2024-I-1",
  "Problem": "Problem statement...",
  "Solution": "Detailed solution...",
  "Answer": "Numerical answer"
}

The model is instructed to solve the problem step by step and provide the final answer in the format “ANSWER: $ANSWER”.

Scoring

The benchmark uses exact match scoring to determine if the model’s answer matches the expected answer.

Results

Model	Accuracy	Std Error
openai/gpt-4o	0.1000	0.0557
openai/gpt-4o-mini	0.0667	0.0463
anthropic/claude-3-5-sonnet-20240620	0.1000	0.0557
anthropic/claude-3-7-sonnet-20250219	0.2000	0.0743

Changelog

[5-A] - 2026-08-04

• Fix scorer only checking the literal last line of the completion, missing answers boxed inside a closing display-math block (e.g. a trailing $$ after \boxed{70}). The scorer no longer mutates state.output.completion, preserving the model’s full raw completion in the persisted eval log.

[4-A] - 2026-05-11

• Fix scorer crash on empty completion (e.g. when model hits max_tokens mid-thought); return INCORRECT instead of raising IndexError.

[3-A] - 2026-03-18

• Unify scorer with AIME 2025 and 2026: add last-line extraction and \boxed{} de-boxing via shared aime_common module.

[2-A] - 2026-02-16

• Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

• Adds backoff policy for functions that connect to huggingface servers.