WorldSense: Grounded Reasoning Benchmark

Measures grounded reasoning over synthetic world descriptions while controlling for dataset bias. Includes three problem types (Infer, Compl, Consist) and two grades (trivial, normal).

Contributed By

@mjbroerman

Code

src/inspect_evals/worldsense

Paper

https://arxiv.org/pdf/2311.15930

Overview

WorldSense is a benchmark to measure reasoning over a world-model while controlling for dataset bias.

Dataset

Here is an example prompt (“description”) from the dataset (problemname == "Compl.trivial"):

Alice is enrolled in 3 courses per week: Alice takes history before computer science and economics before history.
Choose one of the following alternatives: (1) Alice takes history in between economics and computer science, (2) Alice takes history outside of the time range between economics and computer science, or (3) it is impossible to decide.
Think carefully, and only respond with one of these possible options (1), (2), or (3).

There are three problem types:

“Infer” (inference): Determine whether a given statement about a description is true or false.
“Compl” (completion): Select which of three statements are true about a description, including an option for when it is not possible to decide.
“Consist” (consistency): Determine whether a statement about a description is possible or impossible.

In addition there are two grades:

“trivial”: can be solved on statements alone
“normal”: requires a world model to solve

A problemname is formed by concatenation “{type}.{grade}”.

Scoring

Simple accuracy
Standard error
Weighted accuracy by tuple_ID, problemname, and problemsize (as reported in the paper)
Weighted bias by tuple_ID, problemname, and problemsize (as reported in the paper)

In addition to built-in metrics, the main results are weighted accuracy and bias. Here the primary unit is a tuple_ID which corresponds to one “description” or scenario above. To this, up to three answer option sets are provided to ensure that all possible correct answers for the options are selected exactly once. Weights are used to average over multiple tuple_IDs.

All answer options are coded as positive or negative (in the above example, 1: 1, 2: 1, and 3: -1), and all option sets have the same arrangement of option codings. Bias is calculated from this, weighted accordingly.

Problem size corresponds to the number of entities in the description. In the example above, problemsize == 3. Within a problemname, the score is the grand average of problem sizes. If multiple problemnames are specified (including none specified), the score is the grand average of this also.

Parameters

`worldsense`

problemnames (str | list[str]): A string or list of strings specifying the names of problems or tasks to filter the dataset. If provided, it filters the dataset to samples that contain matching metadata for the specified problem names. (default: [])
shuffle (bool): Whether to shuffle the dataset for evaluation. (default: True)

Changelog

[4-A] - 2026-09-21

Derive the sample id from the dataset’s own unique Key. The previous id, (tuple_ID, problemname), is shared by every trial of a tuple, so filter_duplicate_ids kept one trial per tuple and dropped 46,872 of the 87,048 trials. Because the kept trial is the first in file order, the evaluated set held no sample whose gold answer was FALSE, IMPOSSIBLE or 2, while POSSIBLE rose from 23.1% to 50.0% of the set and TRUE from 15.4% to 33.3%.

[3-A] - 2026-09-18

A completion that does not match the answer pattern now scores INCORRECT with reason="invalid_response_format" rather than NOANSWER, following pattern() upstream in inspect_ai#4629, which attributes an unmatched completion to the model under test. Headline metrics are unaffected: value_to_float maps both NOANSWER and INCORRECT to 0.0, so accuracy and stderr are unchanged. The non-answer moves from the score value to Score.reason; log analysis that counts NOANSWER values should read reason instead. See #2314.

[2-A] - 2026-02-16

Migrate version to new scheme. See #907.

Static checks

Results of inspect-evals-lint at the commit this documentation was built from. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…