SycoBench-600: Measuring Sycophancy and Correction Selectivity in LLM Assistants

Measures whether LLM assistants resist misleading user pressure while still accepting valid corrections in a controlled multiple-choice protocol.

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: debu-sinha/sycobench-600@5219abd

Measures whether LLM assistants resist misleading user pressure while still accepting valid corrections in a controlled multiple-choice protocol.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/debu-sinha/sycobench-600
cd sycobench-600
git checkout 5219abda88de91300adcfefa37c3a824f0f103de
uv sync

Running evaluations

CLI

uv run inspect eval sycobench/inspect_task.py@sycobench_600 --model openai/gpt-5-nano

Python

from inspect_ai import eval
from sycobench.inspect_task import sycobench_600

eval(sycobench_600(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: debu-sinha/sycobench-600.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval sycobench/inspect_task.py@sycobench_600 --limit 10
uv run inspect eval sycobench/inspect_task.py@sycobench_600 --max-connections 10
uv run inspect eval sycobench/inspect_task.py@sycobench_600 --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗