Inspect India Evals: Multilingual Reasoning

Evaluates LLM reasoning across major Indian languages, part of the Inspect India Evals suite for assessing model performance in the Indian context.

Maintained By

@MetaFazer

Code

inspect-india-multilingual

Paper

https://arxiv.org/abs/2607.25375

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: MetaFazer/inspect-india-evals@9c2e0bd

Evaluates LLM reasoning across major Indian languages, part of the Inspect India Evals suite for assessing model performance in the Indian context.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/MetaFazer/inspect-india-evals
cd inspect-india-evals
git checkout 9c2e0bd9d8089444751ede080367dcc4983b9dc2
uv sync

Running evaluations

CLI

uv run inspect eval india_evals/multilingual/task.py@multilingual --model openai/gpt-5-nano

Python

from inspect_ai import eval
from india_evals.multilingual.task import multilingual

eval(multilingual(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: MetaFazer/inspect-india-evals.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval india_evals/multilingual/task.py@multilingual --limit 10
uv run inspect eval india_evals/multilingual/task.py@multilingual --max-connections 10
uv run inspect eval india_evals/multilingual/task.py@multilingual --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗