RACE-H: A benchmark for testing reading comprehension and reasoning abilities of neural models

Reading comprehension tasks collected from the English exams for middle and high school Chinese students in the age range between 12 to 18.

Overview

RACE is a benchmark for testing reading comprehension and reasoning abilities of neural models. It is constructed from Chinese middle and high school examinations.

Here, we implement the evaluation for the high school subset (RACE-H) of the RACE dataset. It is formulated as a multiple choice question task. The goal is to choose one out of four options in response to a question based on the provided article. The questions are designed to not merely be text spans in the article, and hence answering them requires reasoning.

The prompt template is based on the multiple choice template in OpenAI’s simple-evals.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/race_h --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.race_h import race_h
eval(race_h)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/race_h --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/race_h --max-connections 10
uv run inspect eval inspect_evals/race_h --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

race_h

• shuffle (bool): (default: True)

Dataset

Here is an example from the dataset (cropped for brevity):

Article: In a small village in England about 150 years ago, a mail coach was standing on the street. It didn’t come to that village often. People had to pay a lot to get a letter. The person who sent the letter didn’t have to pay the postage, while the receiver had to. ...

Question: The first postage stamp was made ____ .

Options: A. in England B. in America C. by Alice D. in 1910

The model is tasked to choose one of the four options.

Scoring

A simple accuracy is calculated over the datapoints.

Changelog

[2-A] - 2026-02-16

• Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

• Adds backoff policy for functions that connect to huggingface servers.