Inspect India Evals: Multilingual Safety Refusal

Evaluates LLM safety refusal behaviour on harmful prompts across Indian languages, part of the Inspect India Evals suite for assessing model performance in the Indian context.

Maintained By

@MetaFazer

Code

inspect-india-multilingual_safety

Paper

https://arxiv.org/abs/2607.25375

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: MetaFazer/inspect-india-evals@9c2e0bd

Evaluates LLM safety refusal behaviour on harmful prompts across Indian languages, part of the Inspect India Evals suite for assessing model performance in the Indian context.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/MetaFazer/inspect-india-evals
cd inspect-india-evals
git checkout 9c2e0bd9d8089444751ede080367dcc4983b9dc2
uv sync

Running evaluations

CLI

uv run inspect eval india_evals/safeguards/task.py@multilingual_safety --model openai/gpt-5-nano

Python

from inspect_ai import eval
from india_evals.safeguards.task import multilingual_safety

eval(multilingual_safety(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: MetaFazer/inspect-india-evals.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval india_evals/safeguards/task.py@multilingual_safety --limit 10
uv run inspect eval india_evals/safeguards/task.py@multilingual_safety --max-connections 10
uv run inspect eval india_evals/safeguards/task.py@multilingual_safety --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗