VoxParity (development split, file-replay track)

VoxParity tests whether voice agents change their typed tool call when an audible cue (emotional delivery, a second voice, environmental sound, speaker age, disfluency, a masked word, sarcasm) changes the correct action while the transcript stays fixed. This task runs the public development split: 40 of the paper’s 183 scenarios, as 81 audio samples plus 40 transcript-only twins; the other 143 are held out. Scoring is judge-free: the first-turn tool call is matched on tool name and typed arguments, with partial credit. Audio-minus-twin metrics show whether hearing the call changes the action, and a summary script adds the paper’s words-only null test against a Whisper + LLM cascade. Install with uv sync --extra inspect.

Maintained By

@bhavik-mangla

Code

voxparity

Paper

https://arxiv.org/abs/2609.35922v1

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: bhavik-mangla/voxparity-bench@dc0b3ed

VoxParity tests whether voice agents change their typed tool call when an audible cue (emotional delivery, a second voice, environmental sound, speaker age, disfluency, a masked word, sarcasm) changes the correct action while the transcript stays fixed. This task runs the public development split: 40 of the paper’s 183 scenarios, as 81 audio samples plus 40 transcript-only twins; the other 143 are held out. Scoring is judge-free: the first-turn tool call is matched on tool name and typed arguments, with partial credit. Audio-minus-twin metrics show whether hearing the call changes the action, and a summary script adds the paper’s words-only null test against a Whisper + LLM cascade. Install with uv sync --extra inspect.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/bhavik-mangla/voxparity-bench
cd voxparity-bench
git checkout dc0b3eda82a746e8e211e3a4f6e17318052cc285
uv sync

Running evaluations

CLI

uv run inspect eval src/voxparity/voxparity_dev/voxparity_dev.py@voxparity_dev --model openai/gpt-5-nano

Python

from inspect_ai import eval
from voxparity.voxparity_dev.voxparity_dev import voxparity_dev

eval(voxparity_dev(), model="openai/gpt-5-nano")

View logs

Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: bhavik-mangla/voxparity-bench.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/voxparity/voxparity_dev/voxparity_dev.py@voxparity_dev --limit 10 --sample-shuffle
uv run inspect eval src/voxparity/voxparity_dev/voxparity_dev.py@voxparity_dev --max-connections 10
uv run inspect eval src/voxparity/voxparity_dev/voxparity_dev.py@voxparity_dev --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗

Static checks

Results of inspect-evals-lint at the registered commit. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…