AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic

AraTrust evaluates LLM trustworthiness when prompted in Arabic via 522 human-written multiple-choice questions (3 options each) spanning 8 categories: truthfulness, ethics, physical health, mental health, unfairness, illegal activities, privacy, and offensive language, with 34 subcategories. Questions were authored by native Arabic speakers or adapted from exams and existing datasets. Scoring is accuracy of selected answer against a single correct option. The eval code loads the dataset from HuggingFace, filters malformed rows, and scores with exact-match choice grading.

Maintained By

@Raulster24

Code

aratrust

Paper

https://arxiv.org/abs/2403.09017

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: Raulster24/aratrust-inspect@e4b89df

AraTrust evaluates LLM trustworthiness when prompted in Arabic via 522 human-written multiple-choice questions (3 options each) spanning 8 categories: truthfulness, ethics, physical health, mental health, unfairness, illegal activities, privacy, and offensive language, with 34 subcategories. Questions were authored by native Arabic speakers or adapted from exams and existing datasets. Scoring is accuracy of selected answer against a single correct option. The eval code loads the dataset from HuggingFace, filters malformed rows, and scores with exact-match choice grading.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/Raulster24/aratrust-inspect
cd aratrust-inspect
git checkout e4b89df399057ca5ae8f8524418907c4868badcd
uv sync

Running evaluations

CLI

uv run inspect eval src/aratrust/aratrust.py@aratrust --model openai/gpt-5-nano

Python

from inspect_ai import eval
from aratrust.aratrust import aratrust

eval(aratrust(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: Raulster24/aratrust-inspect.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/aratrust/aratrust.py@aratrust --limit 10
uv run inspect eval src/aratrust/aratrust.py@aratrust --max-connections 10
uv run inspect eval src/aratrust/aratrust.py@aratrust --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗

Evaluation Report

Timestamp: July 2026

Commit: e4b89df

Model Accuracy Stderr
openai/gpt-4o 0.935 0.011
openai/gpt-4o-mini 0.900 0.013

Notes:

  • 521 usable samples; 1 of 522 source rows dropped for an empty option.
  • Scores exceed the paper’s GPT-4 (81%), as expected for newer models.