AgentBench: Evaluate LLMs as Agents

A benchmark designed to evaluate LLMs as Agents

Overview

AgentBench is designed to evaluate LLMs as agents. The task agent_bench_os contains the “Operating System” tasks from AgentBench.

Felix Hofstätter did the bigger part of implementing these tasks in inspect. Hanna Gábor fixed some bugs, wrote tests and refactored some of the code.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/agent_bench_os --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.agent_bench import agent_bench_os
eval(agent_bench_os)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

The bash and python code is executed inside a Docker container, so you will need to install Docker Engine in order to run the evaluation.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/agent_bench_os --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/agent_bench_os --max-connections 10
uv run inspect eval inspect_evals/agent_bench_os --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

agent_bench_os

• split (str): Dataset split to use (“dev” or “test”). Defaults to “test”., “dev”: 26 samples with public answers for development, “test”: 119 samples for final evaluation (default: 'test')
• solver (Solver | Agent | None): Optional solver to use for the task. If not provided, uses the default agent. (default: None)
• max_attempts (int): Maximum number of attempts allowed per sample. (default: 1)
• max_messages (int): Maximum number of messages allowed per attempt. (default: 100)
• max_sample_time_mins (int): Maximum time a given sample can run. (default: 10)
• sandbox_type (str): The sandbox provider to use (e.g., “docker”, “k8s”, or a custom registered provider). (default: 'docker')
• sandbox_config (Callable[[str, inspect_ai.dataset.Sample], SandboxEnvironmentSpec] | None): Optional custom function to create sandbox specs. Receives (sandbox_type, sample) and returns a SandboxEnvironmentSpec. If None, uses a default config that preserves each sample’s existing compose file. (default: None)

Dataset

The data originates from this repository. The dev set is stored in data/agent_bench_os.json and the test set is stored in data/agent_bench_os_test.json.

Here are a few example tasks (see the paper appendix for more detailed examples including their outputs):

Example 1

There's a program that writes 'succeed' into /root/log periodically. I only know the interval is 1 second, 2 seconds, or 3 seconds. Please find out the interval. Answer me only the integer number of seconds. (e.g. 1, 2, or 3)

Example 2

I would like to implement the following bash command and expose it in your home directory: entering the "calc" command will enable the calculation of an expression. The expression can include addition, subtraction, multiplication, division, and parentheses. If the absolute error between the calculated answer and the expected answer is less than 1e-5, it will be considered correct. For example, I can calculate the result by entering "calc 2 * (9 / 3)", and the output will be 6.

Example 3

Find out count of linux users on this system who belong to at least 2 groups.

Scoring

In most cases, we simply check if the answer matches the answer given in the data file.

For task IDs 3 and 5, the agent needs to write a command. In these cases we compare the output of the agent-written command with the solution-command.

In the original implementation, the scoring (“judging”) mostly happened by running an example script and comparing the agent’s answer to the result of that script. However, this opens up the possibility for the agent to modify the environment in a way that makes the task easier. For example, it could set up its own program that writes to /root/log instead of inspecting how often the currently running program writes into that file. We kept the example codes for testing and documentation purposes.

Changes to the original implementation

General Changes

• Change in the scoring mechanism mentioned in scoring
• Assigned a task id to each task
• The agent home (/home/agent) is the working directory for the container build (where create.init scripts run), the agent’s shell and the scorer, so init scripts and example solutions that use relative paths refer to the same files. Example and check scripts are run in a login shell under the sandbox’s timeout wrapper, exactly like the agent’s bash tool, so they see the same PATH and, for process-counting tasks, the same process tree.
• HOME is /home/agent too (except in tasks that switch to another user), so ~ in task descriptions, init scripts and the agent’s shell all mean the same directory. Previously the agent ran with HOME=/root while the tasks talked about /home/agent.
• The upstream AgentBench clone’s files, apart from the init scripts the build needs, are blanked right after the clone (overwritten with zeros, keeping their sizes, because dev tasks 8, 9 and 11 count files and disk usage under /home). It contains the reference solutions (bootstrap.json) and the check scripts, and models were found reading them. The check scripts (check/*.py, checking/*.sh and the dev split’s check/0.sh) are vendored in data/scripts and copied into the sandbox by the scorer at scoring time, so the agent cannot inspect or replace them beforehand. The agent runs as root, so this protects against inspection, not against a deliberately hostile agent that tampers with bash or python itself.
• The vendored size-match.py accepts fractional sizes such as 4.0K, which du -h produces and which upstream rejected.

Task Specific Changes

Tasks 0-4

Added newline when “succeed” is written to file.

Task 6

The description of the task is “How many hidden files are in /home? (not including subdirectories)”

We changed the example code from:

ls -a /home | grep \"^\\.\" | wc -l

to:

ls -1Ap /home | grep '^\\.' | grep -v '/$' | wc -l

This was needed because of two reasons:

1. -1 forces one entry per line so the count is correct across environments that may format ls output in multiple columns.
2. -A avoids counting . and ..; and -p appends / to directories, enabling grep -v '/$' to exclude directories so only files are counted.

Task 7

Changed the description from “How many files whose name starts with letter exist in /root?” to “How many files whose name starts with an alphanumeric character exist in /root?” to match the code provided later (ls -1 /root | grep \"^[[:alpha:]]\" | wc -l).

Task 12

The description of the task is “How many files in /home (recursively) were modified in the last 24 hours?”

To make this task deterministic, we modified the timestamps of the files present in the docker container to be outside the 24 hours and created new files only after that. More specifically, we changed the code that is run in the docker setup from:

echo \"This is a random string: wuafjva-niqt12r-j9asd19-afj12-sdmxai\">/home/OSbench\ntouch /home/tmpfile\nmkdir /home/tmpdir\ntouch /home/tmpdir/inner"

to:

find /home -type f -exec touch -d \"2023-03-23 12:00:00\" {} \\;\n echo \"This is a random string: <random-string> \">/home/OSbench\ntouch /home/tmpfile\nmkdir /home/tmpdir\ntouch /home/tmpdir/inner

Task 13

Changed task 13 answer to 4 cores and pinned CPUs with cpuset: "0-3" so nproc reports 4 cores consistently.

Task 19

The description is “Find out count of linux users on this system who belong to at least 2 groups.”

Changed the example code from:

getent passwd | cut -d: -f1 | while read user; do groups $user | cut -d ':' -f 2 | tr -s ' ' | sed 's/^ *//;s/ *$//' | grep -q ' ' && echo $user; done | wc -l

to:

getent passwd | cut -d: -f1 | while read user; do groups $user | cut -d ':' -f 2 | tr -s ' ' | sed 's/^ *//;s/ *$//' | tr ' ' '\\n' | grep -v '^$' | tr '\\' '\n' | wc -l | grep -q -P '[2-9]|\\d{2,}' && echo $user; done | wc -l

This was not strictly necessary, we changed this to be consistent with task 20.

Task 20

The description is “Find out count of users who belong to at least 4 groups.”

We changed the example code from:

getent passwd | cut -d: -f1 | while read user; do groups $user | cut -d ':' -f 2 | tr -s ' ' | sed 's/^ *//;s/ *$//' | tr ' ' '\\n' | grep -v '^$' | wc -l | grep -q -w '4' && echo $user; done | wc -l

to:

getent passwd | cut -d: -f1 | while read user; do groups $user | cut -d ':' -f 2 | tr -s ' ' | sed 's/^ *//;s/ *$//' | tr ' ' '\\n' | grep -v '^$' | tr '\\' '\n' | wc -l | grep -q -P '[4-9]|\\d{2,}' && echo $user; done | wc -l

The original command counted only the users who belonged to exactly 4 groups, excluding users who belonged to more than 4 groups.

Task 21

The example code is find . -type f -name "*.helloworld" | wc -l. An earlier version of this port had dropped the quotes, which only worked because the working directory then contained no .helloworld files for the shell to expand the glob against.

Task 23

Now checks the per-user process limit (ulimit -u) instead of the OS-wide threads max, so the answer doesn’t vary by host. The compose file sets a fixed limit.

ulimit -u

Test split

On the test split the expected answer is derived by running the task’s example solution at scoring time, so a buggy example silently becomes the ground truth. The container build, the agent’s shell and the scorer now all use /home/agent as their working directory and the scorer runs examples in a login shell (see General Changes); on top of that, the tasks below were edited individually. Tasks not listed are unchanged from upstream apart from the /root and /usr path rewrite.

Task 5

Description: “Identify the most active traders (buyers/sellers)” now asks for the single most active trader and to answer with the name only, which is what the string-match check against the reference expects.

Tasks 7 and 8

Description: added that regular files are counted, not symbolic links; models had answered 1714, 862 and 687 for the same PATH depending on whether they followed links, and the reference find ... -type f -executable does not.

Task 11

Init: the init is a list (a script file followed by a command); it was silently skipped, so the echo-love fixture never existed. Description: says the directory should be added to PATH in ~/.bashrc, which is where the check looks; a model that edited /home/agent/.bashrc (the task’s “home”) failed the check, which runs as root.

Task 17

Description: “Tell me the max number of threads in Linux” now names the kernel’s threads-max limit and asks for the integer only.

Task 18

Description: “expose it in your home directory” now says the command may be on PATH or an executable file named calc in the home directory; the check invoked calc bare, so a model that did what the wording said (~/calc) failed. Check: the vendored checking/0.sh prepends the home directory to PATH.

Task 19

Description: “If there is a link or something, count it separately” replaced by the semantics of the check script: symbolic links are followed (a link to a regular file counts, a link to a directory is searched), hidden files count, and the command must print only the number. Check: upstream checking/1.sh iterated over "$dir"/*, which skips hidden entries, so a correct find -L ... -type f implementation failed on /root; the vendored copy sets dotglob. The vendored reference example/1.sh sets shopt -s dotglob for the same reason, so it passes its own check. The check also no longer tests /bin and /dev: /usr/bin/X11 is a symlink to its own directory, so following links recurses until the process limit is hit and the count depends on host load, and /dev/fd mirrors the counting process’s own descriptors. It tests /etc instead.

Task 20

Check: upstream checking/2.sh called exit 0 inside its check function, so the script ended after the first of its six date cases and a tool that only handled 2023-5-1 passed. The vendored copy uses return, so all six cases run. Description: states the accepted date spellings (Y-M-D or Y/M/D, two- or four-digit year, one- or two-digit month and day, interpreted as GNU date -d does), which the check exercises but the task never said.

Task 21

Example: ls -a /home/agent | grep "^\." | wc -l counted . and ... Now ls -1Ap /home/agent | grep '^\.' | grep -v '/$' | wc -l, as for dev task 6.

Task 22

Description: “starts with letter” was read literally by a model (it looked for files named letter*); now “starts with a letter”, which is what the reference grep "^[[:alpha:]]" matches.

Task 25

Description: states that symbolic links to directories (/bin, /lib, /lib64, /sbin) are not counted, as the reference ls -l / | grep "^d" does not count them.

Task 26

Description: states that /home/agent itself is counted, as the reference find /home/agent -type d | wc -l does.

Task 27

Init: converted to a script that runs at container start (create.init_at_start), so the files are always “modified in the last 24 hours” when the agent runs. Previously they were created at image build time and the answer decayed from 3 to 0 once the image was a day old.

Task 35

Check: ls -l / | grep data | awk '{if ($1~/^d.....s..t/) ...' demanded lowercase s and t, i.e. group and world execute bits, although the task says the directory must be accessible only to the group, and it never looked at the group or the other permission bits, so a world-readable /data in the root group passed. Now [ "$(stat -c '%G %a' /data)" = "dataaccess 3770" ]: group dataaccess, setgid, sticky, no access for others. Init: creates the dataaccess group the task refers to. Example: chgrp dataaccess /data && chmod 3770 /data (upstream’s never set the group).

Task 36

Removed. The sandbox has no network, so sudo cannot be reinstalled; the only way to pass the check was to fake a sudo command.

Task 37

Init: the five background sleep processes were started during the image build and did not survive it. They now start at container start and run for the duration of the task (sleep infinity); the description no longer says “for 60 seconds each”. Example: ps -A | wc -l counted the header line and, depending on how the count is invoked, the counting command’s own processes; the description now says not to count those, naming the timeout wrapper the agent’s commands run under, and the reference counts the processes that predate the counting command (PIDs below its timeout wrapper).

Task 38

Description: added “Do not count the shell that runs your own command”, which is what the reference (grep -v grep) effectively does.

Task 39

Removed. The file extension and directory are “given to you as input arguments” that the task never supplies.

Task 40

Description: the word “error” is now stated to be matched case-insensitively, as in the reference grep -iw.

Task 41

Init: the five temp_student processes were started during the image build with sudo, which the image does not have, and would not have survived the build anyway. They now start at container start with runuser -u temp_student -- sleep infinity.

Task 42

Example: find ... -exec wc -l {} + emits a total line when given several files, doubling the sum. Now -exec wc -l {} \;. Description: “text files that reside in both dir1 and dir2” was read by a model as files present in both directories; it now says the two directories’ line counts are added together.

Task 43

Removed. The description is a leaked generation prompt (“Your task is to devise a problem, bash scripts for initialization, and an example script”), and the memory-usage answer changes between the agent’s and the scorer’s measurement.

Task 45

Example: find $TARGET_DIR ... relied on a variable exported only inside the init script. Now find "$HOME/TA_problem" .... Description: states that the directory itself is counted, as the reference does.

Task 46

Example and check: the task asks for TXT_COUNT CSV_COUNT LOG_COUNT but the example summed the three counts into one integer. The example now prints the three counts and the check is string-match.py.

Task 48

Description: states that task_directory and its files already exist and must not be changed, and that each octal mode (e.g. 644) is summed as a decimal number, as the reference does.

Task 51

Removed. The lines to count are those containing “a specified keyword” that the task never specifies.

Task 52

Description: the upstream text is cut off mid-sentence (“…corresponding index in the range”). Completed from the reference solution: files contain their index, and the answer is the sum of the odd-indexed files.

Task 53

Example: ls -l | grep -v "^d\| \." was meant to drop hidden entries but also dropped the visible files. Now find experiment_directory -mindepth 1 -maxdepth 1 -not -name '.*' | wc -l.

Task 55

Removed. The answer sums a process count, a memory percentage and a disk percentage; the memory term fluctuates between measurements, so the agent’s and the scorer’s values differ by design.

Task 56

Example: grep -o over several files prefixes each match with its filename, so the same IP address in two files was counted twice. Now grep -oh.

Tasks 57, 62, 71, 85 and 113

Description: these tasks tell the agent to build the fixture itself; they now say to build it in the current working directory, where the reference solution looks for it.

Task 59

Description: “total size (in kilobytes)” now says disk usage in kilobytes (allocated space in 1K blocks, not the apparent size), which is what the reference measures.

Task 62

Example: the task asks for the sum of the sizes of all files and directories inside test_directory, but the reference summed only files (-type f). Now find test_directory -mindepth 1 -exec du -b {} +.

Task 63

Description: “Ignore any empty lines. Only count lines with actual code” now also says that comment lines (starting with #) are ignored, as in the reference.

Task 66

Example: find . from the working directory counted the parent directory that the task says to exclude. Now find problem_directory -mindepth 1 | wc -l.

Task 68

Example: ls -a ~ | grep '^\.' counted hidden files (.bashrc, .profile) as hidden directories. Now find ~ . -mindepth 1 -maxdepth 1 -type d -name '.*' -exec realpath {} + | sort -u | wc -l: the home and the working directory are the same directory (/home/agent), so the two counts are no longer added; the description asks for distinct directories and says to count each once.

Task 70

Example: counted project_directory itself although the task asks for the entries inside it. Now find project_directory -mindepth 1 | wc -l.

Task 74

Description: states that /var/log itself is counted, as the reference does.

Task 76

Example: xargs -0 wc -l emits a total line, doubling the sum. Now -exec wc -l {} \;.

Task 78

Description: states that exp_folder itself is counted, as the reference does.

Task 79

Example: du -ckB1 reported bytes although the task asks for kilobytes. Now du -ck.

Task 79 (description)

Description: “total size in kilobytes” now says disk usage in kilobytes (allocated space, not the apparent size), as for task 59; the apparent size of the five files is 73 bytes, which rounds to 0.

Task 86

Description: the extension was “provided as an argument (e.g.,”.txt”)” that the task never supplies; it is now .txt, which the reference counts.

Task 91

Example: wc -l {} + total line, as for task 42. Now -exec wc -l {} \;.

Task 93

Example: the three wc invocations used {} + and doubled their sums. Now {} \;. Description: states that the directory itself is counted among the folders, as the reference find . -type d does.

Task 94

Example: the init copies test_environment to student_directory, so find . from the working directory summed both copies. Now find student_directory ....

Task 96

Example: counted /home/agent itself although the task asks for its subdirectories. Now find /home/agent -mindepth 1 -maxdepth 2 -type d | wc -l.

Task 97

Example: grep -c on several files prints file:count, so awk '{sum += $1}' summed zeros. Now awk -F: '{sum += $NF}'.

Task 101

Example: wc -l {} + total line, as for task 42. Now -exec wc -l {} \;.

Task 106

Description: states that the directory itself is counted, as the reference find ~/testDir -type f -o -type d | wc -l does.

Task 110

Init: the Python files were written with echo "...\n..." and so contained a literal \n instead of newlines, making the “lines of code” answer 4 for files that were clearly meant to have 13 lines. Now echo -e.

Task 111

Example: find /etc -maxdepth 1 -mindepth 1 -type f -o -type d excluded the symlinks in /etc although the task asks for all files and directories. Now find /etc -mindepth 1 -maxdepth 1 | wc -l.

Task 112

Example: ps -e | wc -l counted the header line and the counting command’s own processes; the description now says not to count those, naming the timeout wrapper, and the reference counts the processes that predate the counting command (PIDs below its timeout wrapper), as for task 37.

Task 114

Example: wc -lwm {} + total line doubled all three sums. Now -exec wc -lwm {} \;.

Task 114 (description)

Description: “Your final answer should be in the format: L + W + C” made a model submit 4 + 13 + 83; it now asks for the sum as a single integer, which is what the reference prints.

Task 115

Example: -type d -o -type f -not -path "*/\.*" bound -not to files only, so hidden directories were counted but hidden files skipped although the task includes hidden entries. Now find ~/example_directory \( -type d -o -type f \) | wc -l.

Task 117

Example: grep -r -c 'Linux' ~/Linux_files/*.txt never reached the subdirectory the task includes. Now grep -r -c 'Linux' --include='*.txt' ~/Linux_files.

Task 118

Example: find ~/TA_problem_directory -maxdepth 1 | wc -l | awk '{print $1-2}' subtracted 2 from a count that includes the directory once, giving 4 for five entries. Now find ~/TA_problem_directory -mindepth 1 -maxdepth 1 | wc -l. Description: names the directory and says “directly inside, without descending into subdirectories” instead of “excluding its subdirectories”.

Task 119

Example: wc -l {} + total line, as for task 42. Now -exec wc -l {} \;.

Task 120

Example: the server id and the date were parsed from the wrong _-separated fields because the path included the log_files/ prefix, so no file ever matched. Now basename is applied first, the id is the third field and the date the last.

Task 124

Description: “total file size in kilobytes” now says apparent file sizes rather than disk usage, rounded down to whole kilobytes, which is what the reference (du -sb --apparent-size) computes; a model reporting disk usage (du -k, 28) instead of apparent size (15) failed.

Task 121

Removed. The user id is “provided as an input argument” that the task never supplies; with it empty the reference always printed 0.

Comparison to the results in the original paper

The reported success rate for gpt-4 is \(42.4\% \; (N=144)\). Running this implementation with gpt-4 only on dev data, the success rate is \(69.2\% \pm 9.2\% \; (N=26)\). An earlier version of this port scored gpt-4 at \(36\% \pm 9.8\% \; (N=25)\) on the test split, but that figure was produced by the broken ground truth fixed in version 4 (see Test split) and should not be compared with anything. On version 4 the test split is considerably easier than the paper’s numbers suggest: glm-5.3-flash scored \(95.8\% \pm 1.8\%\) and deepseek-v4-flash-0731 \(97.5\% \pm 1.4\%\) \((N=119)\) on the final version 4 tasks (full runs, with the tasks whose wording changed afterwards rerun and spliced in) (earlier runs during the fix, before the check-script and reference-material changes, gave \(87.4\%\) and \(90.8\%\)), and the same models’ logged answers rescored against the corrected expectations give similar numbers, so most of the paper-era difficulty was the tasks being unsolvable rather than hard.

To-Do

• We kept the current path-rewrite approach in recursively_set_agent_home() (blanket string replacement of /root and /usr to /home/agent) as-is for now. See this PR comment.

Changelog

[4-B] - 2026-09-02

• Added sandbox_type and sandbox_config parameters for provider flexibility (sandbox-config pattern, #1115).

[4-A] - 2026-09-12

• Fixed the test split ground truth (see #2420): task fixtures were built in /home while the scorer derived the expected answer from /home/agent, so every task with a relative-path init script was unsolvable. The scorer now runs example and check scripts in a login shell from the agent home, the upstream size check accepts du -h output, and twenty-seven buggy example solutions were corrected (see Test split). Blank lines in create.init.code no longer produce an empty RUN instruction. Tasks 27, 37 and 41 run their init at container start so their fixtures exist when the agent runs, and tasks 36, 39, 43, 51, 55 and 121 were removed because they have no well-defined answer (see Test split); the split now has 119 tasks. Ambiguous task descriptions were clarified to match their reference solutions (see Test split), and the system prompt now tells the agent to submit only the final answer (a bare integer or the exact string) and, when a task asks for a script, to run it and submit its output. The previous prompt only asked for a bare integer when the answer was one, and models regularly submitted the script instead. HOME is now /home/agent like the working directory. The upstream repository clone, which held the reference solutions and check scripts and which models were found reading, is blanked in the image; the check scripts are vendored and staged by the scorer at scoring time, with three of them corrected: task 20’s ran only the first of six cases, task 19’s skipped hidden files, and task 35’s accepted a world-readable directory in the wrong group while rejecting the correct mode.

[3-A] - 2026-04-21

• Replaced deprecated basic_agent() with react() as the default agent. This fixes a pathological case where basic_agent() would loop on repeated content_filter stops until hitting message_limit. It also changes the stall-nudge message (sent when the model stops without calling a tool) to append a reminder to call the submit tool — a one-sentence prompt addition that may bias models toward earlier submission.

[2-A] - 2026-02-16

• Migrate version to new scheme. See #907.

Static checks

Results of inspect-evals-lint at the commit this documentation was built from. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…