AppWorld Benchmark (Interactive Coding Agents over App APIs)

AppWorld evaluates autonomous agents on 750 day-to-day digital tasks requiring iterative Python code generation against 457 APIs across 9 simulated apps. Tasks are split into test_normal (168) and test_challenge (417, includes unseen Amazon/Gmail APIs). Scoring uses programmatic state-based unit tests checking database diffs for goal completion (Task Goal Completion, TGC) and collateral damage avoidance, rather than reference-solution comparison. Scenario Goal Completion (SGC) measures consistency across task variants.

Maintained By

@anirudhvenu

Code

appworld

Paper

https://arxiv.org/abs/2407.18901v1

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: anirudhvenu/appworld-inspect@880fbc4

AppWorld evaluates autonomous agents on 750 day-to-day digital tasks requiring iterative Python code generation against 457 APIs across 9 simulated apps. Tasks are split into test_normal (168) and test_challenge (417, includes unseen Amazon/Gmail APIs). Scoring uses programmatic state-based unit tests checking database diffs for goal completion (Task Goal Completion, TGC) and collateral damage avoidance, rather than reference-solution comparison. Scenario Goal Completion (SGC) measures consistency across task variants.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/anirudhvenu/appworld-inspect
cd appworld-inspect
git checkout 880fbc47e73270576137b928fdb1ecab61529da1
uv sync

Running evaluations

CLI

uv run inspect eval src/appworld_inspect/appworld_inspect.py@appworld --model openai/gpt-5-nano

Python

from inspect_ai import eval
from appworld_inspect.appworld_inspect import appworld

eval(appworld(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: anirudhvenu/appworld-inspect.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/appworld_inspect/appworld_inspect.py@appworld --limit 10
uv run inspect eval src/appworld_inspect/appworld_inspect.py@appworld --max-connections 10
uv run inspect eval src/appworld_inspect/appworld_inspect.py@appworld --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗