SALAD-Bench: Hierarchical Safety Benchmark (Multiple-Choice)

The multiple-choice half of SALAD-Bench, a hierarchical LLM safety benchmark spanning 6 top-level safety domains, 16 task categories and 65 fine-grained categories. Each of the 3,832 scorable questions pairs a potentially harmful query with three candidate answers, and the model must select the safe option(s). One or two of the three may be safe, so the gold label is a set and scoring is an exact set match: selecting only one safe option when two are safe is wrong, as is including an unsafe option. Scoring is deterministic and needs no judge model, using SALAD-Bench’s regex MC-Judge over the <ans>...<eoa> envelope (as distinct from the model-based MD-Judge used for the free-form QA sets). Reports accuracy and stderr overall, plus accuracy grouped by top-level safety domain.

Maintained By

@WatchTree-19

Code

salad-bench

Paper

https://arxiv.org/abs/2402.05044

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: WatchTree-19/inspect-salad-bench@e85b9fe

The multiple-choice half of SALAD-Bench, a hierarchical LLM safety benchmark spanning 6 top-level safety domains, 16 task categories and 65 fine-grained categories. Each of the 3,832 scorable questions pairs a potentially harmful query with three candidate answers, and the model must select the safe option(s). One or two of the three may be safe, so the gold label is a set and scoring is an exact set match: selecting only one safe option when two are safe is wrong, as is including an unsafe option. Scoring is deterministic and needs no judge model, using SALAD-Bench’s regex MC-Judge over the <ans>...<eoa> envelope (as distinct from the model-based MD-Judge used for the free-form QA sets). Reports accuracy and stderr overall, plus accuracy grouped by top-level safety domain.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/WatchTree-19/inspect-salad-bench
cd inspect-salad-bench
git checkout e85b9feb8d024890874694beb32e4bbf4564f169
uv sync

Running evaluations

CLI

uv run inspect eval src/inspect_salad_bench/salad_bench.py@salad_bench_mcq --model openai/gpt-5-nano

Python

from inspect_ai import eval
from inspect_salad_bench.salad_bench import salad_bench_mcq

eval(salad_bench_mcq(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: WatchTree-19/inspect-salad-bench.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/inspect_salad_bench/salad_bench.py@salad_bench_mcq --limit 10
uv run inspect eval src/inspect_salad_bench/salad_bench.py@salad_bench_mcq --max-connections 10
uv run inspect eval src/inspect_salad_bench/salad_bench.py@salad_bench_mcq --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗