RACE-H: A benchmark for testing reading comprehension and reasoning abilities of neural models

Reading comprehension tasks collected from the English exams for middle and high school Chinese students in the age range between 12 to 18.

Overview

RACE is a benchmark for testing reading comprehension and reasoning abilities of neural models. It is constructed from Chinese middle and high school examinations.

Here, we implement the evaluation for the high school subset (RACE-H) of the RACE dataset. It is formulated as a multiple choice question task. The goal is to choose one out of four options in response to a question based on the provided article. The questions are designed to not merely be text spans in the article, and hence answering them requires reasoning.

The prompt template is based on the multiple choice template in OpenAI’s simple-evals.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/race_h --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.race_h import race_h
eval(race_h)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/race_h --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/race_h --max-connections 10
uv run inspect eval inspect_evals/race_h --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

race_h

• shuffle (bool): (default: True)

Dataset

Here is an example from the dataset (cropped for brevity):

Article: In a small village in England about 150 years ago, a mail coach was standing on the street. It didn’t come to that village often. People had to pay a lot to get a letter. The person who sent the letter didn’t have to pay the postage, while the receiver had to. ...

Question: The first postage stamp was made ____ .

Options: A. in England B. in America C. by Alice D. in 1910

The model is tasked to choose one of the four options.

Scoring

A simple accuracy is calculated over the datapoints.

Known dataset issues

The all config’s train split contains two duplicated records, reported at ehovy/race discussion #4 and still open at the pinned revision. This task evaluates the high config’s test split, which has no repeated records, so the port no longer deduplicates.

Changelog

[2-A] - 2026-02-16

• Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

• Adds backoff policy for functions that connect to huggingface servers.

Static checks

Results of inspect-evals-lint at the commit this documentation was built from. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…