BrokenMath: Sycophancy in LLM Theorem Proving

BrokenMath measures sycophancy in LLMs by presenting adversarially falsified recent olympiad theorems and asking models to prove them. This eval runs the dataset’s benchmark split — 451 adversarial, proof-style problems (perturbed from 2025 competition sources and verified by an IMO medalist). A 4-way LLM-as-judge classifies each response as correct/detected/corrected/incorrect; the primary metric is the sycophancy rate (fraction judged incorrect — the model attempted a proof of the false statement without flagging the flawed premise).

Maintained By

@Vedant-Agarwal

Code

brokenmath

Paper

https://arxiv.org/abs/2510.04721v1

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: Vedant-Agarwal/inspect-brokenmath@d37b80b

BrokenMath measures sycophancy in LLMs by presenting adversarially falsified recent olympiad theorems and asking models to prove them. This eval runs the dataset’s benchmark split — 451 adversarial, proof-style problems (perturbed from 2025 competition sources and verified by an IMO medalist). A 4-way LLM-as-judge classifies each response as correct/detected/corrected/incorrect; the primary metric is the sycophancy rate (fraction judged incorrect — the model attempted a proof of the false statement without flagging the flawed premise).

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/Vedant-Agarwal/inspect-brokenmath
cd inspect-brokenmath
git checkout d37b80bb16c9d98df990a7edd92323a1d73822bb
uv sync

Running evaluations

CLI

uv run inspect eval src/brokenmath/brokenmath.py@brokenmath --model openai/gpt-5-nano

Python

from inspect_ai import eval
from brokenmath.brokenmath import brokenmath

eval(brokenmath(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: Vedant-Agarwal/inspect-brokenmath.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/brokenmath/brokenmath.py@brokenmath --limit 10
uv run inspect eval src/brokenmath/brokenmath.py@brokenmath --max-connections 10
uv run inspect eval src/brokenmath/brokenmath.py@brokenmath --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗