FRAMES: Factual Retrieval and Multi-hop Evaluation Set (baseline variant)

FRAMES evaluates multi-hop reasoning by requiring models to answer questions that can only be resolved by chaining information across multiple Wikipedia documents. In the baseline variant, all source documents are provided upfront, testing whether a model can reason across a full document set to produce the correct answer. Scored on answer accuracy.

Maintained By

@sahil350

Code

frames_baseline

Paper

https://arxiv.org/abs/2409.12941

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: sahil350/frames-eval@2fbb1a1

FRAMES evaluates multi-hop reasoning by requiring models to answer questions that can only be resolved by chaining information across multiple Wikipedia documents. In the baseline variant, all source documents are provided upfront, testing whether a model can reason across a full document set to produce the correct answer. Scored on answer accuracy.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/sahil350/frames-eval
cd frames-eval
git checkout 2fbb1a123431d559180547e325f0765d97ab2a39
uv sync

Running evaluations

CLI

uv run inspect eval src/frames_eval/frames.py@frames_baseline --model openai/gpt-5-nano

Python

from inspect_ai import eval
from frames_eval.frames import frames_baseline

eval(frames_baseline(), model="openai/gpt-5-nano")

View logs

Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: sahil350/frames-eval.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/frames_eval/frames.py@frames_baseline --limit 10 --sample-shuffle
uv run inspect eval src/frames_eval/frames.py@frames_baseline --max-connections 10
uv run inspect eval src/frames_eval/frames.py@frames_baseline --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗

Evaluation Report

Timestamp: May 2026

Commit: 2fbb1a1

frames_baseline

Model Accuracy
ollama/llama3.2 0.060
ollama/llama3.1:8b 0.080

Notes:

  • Wikipedia content fetched live via the MediaWiki API at eval time.

Static checks

Results of inspect-evals-lint at the registered commit. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…