PerspectiveGap Role-Fragment Assignment

Evaluates LLMs’ ability to assign information fragments to sub-agent roles in multi-agent orchestration scenarios. Each of 110 scenarios provides a role list, shuffled labeled fragments (7-13), and one distractor; the model outputs a JSON mapping each role to needed fragment IDs. Scoring uses strict pass (zero omissions and zero leaks) plus partial-credit metrics. Data covers 10 loop-centered topologies across 100 professional domains, validated via a 716-row hand-audited scorer test set.

Maintained By

@WhymustIhaveaname

@guojx1998

Code

perspective_gap

Paper

https://arxiv.org/abs/2606.08878v1

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: WhymustIhaveaname/PerspectiveGap-inspect@9ebdf21

Evaluates LLMs’ ability to assign information fragments to sub-agent roles in multi-agent orchestration scenarios. Each of 110 scenarios provides a role list, shuffled labeled fragments (7-13), and one distractor; the model outputs a JSON mapping each role to needed fragment IDs. Scoring uses strict pass (zero omissions and zero leaks) plus partial-credit metrics. Data covers 10 loop-centered topologies across 100 professional domains, validated via a 716-row hand-audited scorer test set.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/WhymustIhaveaname/PerspectiveGap-inspect
cd PerspectiveGap-inspect
git checkout 9ebdf214922cf6d2f2306d03b9f3b4569e496703
uv sync

Running evaluations

CLI

uv run inspect eval src/perspective_gap_inspect/role_assignment.py@role_assignment --model openai/gpt-5-nano

Python

from inspect_ai import eval
from perspective_gap_inspect.role_assignment import role_assignment

eval(role_assignment(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: WhymustIhaveaname/PerspectiveGap-inspect.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/perspective_gap_inspect/role_assignment.py@role_assignment --limit 10
uv run inspect eval src/perspective_gap_inspect/role_assignment.py@role_assignment --max-connections 10
uv run inspect eval src/perspective_gap_inspect/role_assignment.py@role_assignment --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗