The Agent Company: Evaluating multi-tool autonomous agents in a synthetic company

The Agent Company benchmark evaluates autonomous agents in a realistic, self-contained company environment. Tasks require browsing internal web services, reading and writing files, running code, and coordinating tools to solve multi-step problems.

Contributed By

@bndxn

Code

src/inspect_evals/theagentcompany

Paper

https://arxiv.org/abs/2412.14161

Overview

This eval provides an adaptation of the original TheAgentCompany implementation. Where possible it reuses code with attribution.

The full eval comprises 175 different tasks that connect to a range of bespoke services. To make this implementation easier to review, this work is split into stages:

  1. Basic example tasks - 5 tasks, ownCloud only - to check approach
  2. Basic remaining - 28 tasks, ownCloud only - splitting to make the review easier
  3. Supporting non-player LLM interactions - 5 tasks, adding Rocketchat - check approach
  4. Remaining non-player LLM interactions - about 50
  5. Coding tasks - 5 tasks, gitlab
  6. All remaining tasks - 87 tasks

The first two PRs have been merged. The current PR focuses on step 3 (Rocket.Chat connectivity and LLM-driven NPC colleagues).

Usage

Installation

This eval requires extra dependencies.

pip install inspect-evals[theagentcompany]

From a checkout of this repository:

uv sync --extra theagentcompany

Running evaluations

uv run inspect eval inspect_evals/theagentcompany --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.theagentcompany import theagentcompany
eval(theagentcompany)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Pre-flight checks (before running agents)

Structural checks (dataset YAML, asset paths, scorer imports, optional Docker Compose validation) without a live model:

uv sync --extra theagentcompany
uv run pytest tests/theagentcompany/

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/theagentcompany --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/theagentcompany --max-connections 10
uv run inspect eval inspect_evals/theagentcompany --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

theagentcompany

  • task_name (Union[Literal['admin_check_employees_budget_and_reply', 'admin_make_spreadsheet', 'admin_mass_forms_filling', 'admin_remove_pages_pdf', 'admin_translate_sales_chat', 'admin_watch_video', 'ds_answer_numerical_data_question', 'ds_coffee_shop_database_management', 'ds_find_meeting_spreadsheet', 'ds_fix_table_values_and_missing_answers', 'ds_format_excel_sheets', 'ds_predictive_modeling', 'ds_sql_exercise', 'ds_stock_analysis_slides', 'ds_visualize_data_in_pie_and_bar_chart', 'finance_apply_tax_credit', 'finance_budget_variance', 'finance_check_attendance_payroll', 'finance_create_10k_income_report', 'finance_expense_validation', 'finance_invoice_matching', 'finance_nonqualified_bill_ask_for_reimburse', 'hr_check_attendance_multiple_days', 'hr_check_attendance_multiple_days_department', 'hr_check_attendance_multiple_days_department_with_chat', 'hr_check_attendance_one_day', 'hr_create_employee_manual', 'hr_green_card_consultation', 'hr_new_grad_job_description', 'hr_organize_talent_info', 'hr_populate_salary_increase_memo', 'hr_resume_categorization', 'hr_salary_analysis', 'ml_generate_gradcam', 'ml_grade_exam', 'research_answer_questions_on_paper', 'sde_copy_table_from_pdf_to_xlsx', 'sde_create_sqlite_database', 'sde_run_rising_wave_locally'], Literal['test_basic_llm_judge', 'test_owncloud_file_reading', 'test_install_permissions', 'test_evaluator_helpers', 'test_rocketchat_load_npcs', 'test_rocketchat_npc_reply'], None]): The name of the task to load. Can be a real task or a test task. If omitted, runs all real tasks and dynamically selects the correct scorer per sample via the sample’s eval_name metadata. (default: None)
  • scoring_mode (Literal['original', 'improved']): For ds_sql_exercise, choose original (exact file match) or improved (LLM-as-judge on the same outputs). Other tasks ignore this and use their usual scorer. (default: 'original')
  • max_turns (int): Maximum agent turns (model generations). Matches the original TheAgentCompany OpenHands max_iterations=100 (not a message count). (default: 100)
  • npc_model (str | None): Model ID for LLM-driven NPC colleagues. The original paper backs all NPCs with anthropic/claude-3-5-sonnet-20241022; pass that for comparability. Only used for tasks whose eval.yaml declares an npcs metadata block. If omitted, those tasks raise an error. (default: None)

Running individual tasks

To run a specific task, pass task_name via -T:

# Run a specific task
uv run inspect eval inspect_evals/theagentcompany -T task_name=hr_populate_salary_increase_memo --model anthropic/claude-haiku-4-5

Some tasks also support an LLM-judge scoring mode:

uv run inspect eval inspect_evals/theagentcompany -T task_name=ds_sql_exercise -T scoring_mode=improved --model openai/gpt-5-nano --limit 1

Dataset

This eval provides a self-contained environment with internal websites and data. To replicate those systems, it uses additional service containers. For example, a task might require the agent to use a web browser to access a company website served by a connected container.

A simpler task is to download a spreadsheet from a web server, read the file, perform calculations, and save the result in the workspace. More complex tasks can involve interacting with colleagues, installing software, and running local tooling.

There is also a small set of test tasks, provided to help development and to test that the environment and agent is configured correctly: test_basic_llm_judge, test_owncloud_file_reading, test_install_permissions, test_evaluator_helpers.

Scoring

Scoring is done using a custom scorer for each task, which runs on the host. All tasks implemented so far have an original scoring method, which matches the original implementation as closely as possible.

However, many of these scorers have issues - for example relying on string matching e.g. ds_answer_numerical_data_question, and therefore failing correct answers due to minor formatting differences. Others fail models for not achieving results not specified in the eval, for example ml_grade_exam, or use very stringent marking criteria research_answer_questions_on_paper. In these cases, an improved scoring method has been implemented, access using the parameter -T scoring_mode=improved.

Metrics

Each task reports two kinds of outcome in a single Score.value dict:

  • score — overall pass or fail (C or I), matching the original TheAgentCompany “all checkpoints passed” rule.
  • checkpoints — partial progress as a float from 0.0 to 1.0 (for example 0.2 means roughly one in five checkpoints passed).

That split is deliberate: you get a clear pass/fail for leaderboard-style accuracy, and a separate signal for how far the agent got when it did not fully succeed.

Inspect is wired to treat each field on its own. TAC_SCORER_METRICS maps metrics to dict keys so accuracy() runs on score and checkpoint averaging runs on checkpoints, rather than on the whole dict. In the eval log you should see both metrics (sometimes under separate entries named score and checkpoints).

The react solver also needs to know whether a submission passed. After each submit(), it checks the score to decide whether to retry. theagentcompany.py passes tac_attempt_score_value, which reads only the score field for that decision—the same pass/fail semantics as the metrics, without mixing in checkpoint progress.

If you still see a warning like unable to convert value to float: {'score': 'C', 'checkpoints': 1.0}, Inspect received the full dict where it expected a single scalar. That usually means an older checkout, a log rescored before this layout, or a custom integration that has not adopted the dict-key metrics. Re-running the eval on current code is the right fix; the run itself is otherwise fine.

Evaluation Report

The Agent Company team maintains a leaderboard of model performance. The table below reproduces figures from their published run OpenHands–Versa with Claude Sonnet 4 (evaluation 1.0.0, 14 June 2025).

Report configuration:

  • Eval version: 3-B
  • A model ID (published baseline): anthropic/claude-sonnet-4
  • B model ID (this implementation): anthropic/claude-haiku-4-5-20251001

Commands used to produce the B results in this table:

uv run inspect eval inspect_evals/theagentcompany --model anthropic/claude-haiku-4-5-20251001 --limit 1
uv run inspect eval inspect_evals/theagentcompany -T task_name=ds_sql_exercise -T scoring_mode=improved --model anthropic/claude-haiku-4-5-20251001 --limit 1
uv run inspect eval inspect_evals/theagentcompany -T task_name=research_answer_questions_on_paper -T scoring_mode=improved --model anthropic/claude-haiku-4-5-20251001 --limit 1

By default the same model as the agent will be used as the grader, for tasks requiring a grader model. If you would like to specify a different grader model, you can do this using this syntax:

uv run inspect eval inspect_evals/theagentcompany --model anthropic/claude-haiku-4-5-20251001 --limit 1 --model-role grader=anthropic/claude-haiku-4-5-20251001

The Agent Company leaderboard results (⭐ indicates perfect completion)

This section contains a comparison with results from the original implementation, available in this folder. I spent a while trying to set up the original implementation on a Mac but was not able to, so the results from this leaderboard are used. One of the most capable models they used was Claude Sonnet 4. At the time of writing this is no longer available, so I use Claude Haiku 4.5 as it has similar performance and lower cost.

A — OpenHands–Versa with Claude Sonnet 4 (anthropic/claude-sonnet-4, published run linked above). B — this implementation with Claude Haiku 4.5 (anthropic/claude-haiku-4-5-20251001). The grader model for this implementation is the same as the agent model: Claude Haiku 4.5 (anthropic/claude-haiku-4-5-20251001).

Extracted task scores for implemented tasks from the OpenHands–Versa Claude Sonnet 4 results README (original implementation) and from TRAJECTORY_ANALYSIS.md / stage-3 Haiku analysis (checkpoint scores for Inspect implementation):

Task Original implementation Inspect implementation
ds-fix-table-values-and-missing-answers-image 1.00 0.67
ds-sql-exercise-image 1.00 0.67
finance-check-attendance-payroll-image 1.00 0.33
hr-check-attendance-one-day-image 1.00 1.00
hr-new-grad-job-description-image 1.00 0.00
sde-create-sqlite-database-image 0.44 0.88
finance-expense-validation-image 0.38 0.50
hr-green-card-consultation-image 0.33 0.33
admin-check-employees-budget-and-reply-image 0.25 0.33
ds-find-meeting-spreadsheet-image 0.25 0.50
ds-visualize-data-in-pie-and-bar-chart-image 0.25 0.50
finance-create-10k-income-report-image 0.25 0.80
hr-check-attendance-multiple-days-image 0.25 0.67
admin-remove-pages-pdf-image 0.17 0.67
hr-populate-salary-increase-memo-image 0.14 0.83
finance-budget-variance-image 0.12 1.00
hr-check-attendance-multiple-days-department-with-chat-image 0.12 -
hr-create-employee-manual-image 0.12 0.33
hr-organize-talent-info-image 0.12 0.25
hr-resume-categorization-image 0.12 0.25
admin-make-spreadsheet-image 0.00 0.00
admin-mass-forms-filling-image 0.00 0.00
admin-translate-sales-chat-image 0.00 0.00
admin-watch-video-image 0.00 0.00
ds-answer-numerical-data-question-image 0.00 0.00
ds-coffee-shop-database-management-image 0.00 0.00
ds-format-excel-sheets-image 0.00 0.00
ds-predictive-modeling-image 0.00 1.00
ds-stock-analysis-slides-image 0.00 0.17
finance-apply-tax-credit-image 0.00 0.00
finance-invoice-matching-image 0.00 0.33
finance-nonqualified-bill-ask-for-reimburse-image 0.00 0.00
hr-check-attendance-multiple-days-department-image 0.00 0.00
hr-salary-analysis-image 0.00 0.00
ml-generate-gradcam-image 0.00 0.00
ml-grade-exam-image 0.00 0.00
research-answer-questions-on-paper-image 0.00 0.58
sde-copy-table-from-pdf-to-xlsx-image 0.00 0.20
sde-run-rising-wave-locally-image 0.00 1.00

Scoring and multiple epochs

Task scorers store the aggregate checkpoint completion rate in Score.value["checkpoints"] so Inspect can reduce it correctly when running with --epochs greater than 1. Per-checkpoint detail remains in Score.metadata["checkpoints"] for log inspection; that metadata is not averaged across epochs.

Testing

This eval includes several test tasks that should pass quickly for most capable models. They verify that services are connected and required data is loaded.

  • test-owncloud-file-reading - verifies ownCloud data is loaded and accessible via the browser
  • test-basic-llm-judge - verifies an LLM judge can evaluate container-provided content
  • test-install-permissions - verifies the agent can install packages in the sandbox (poppler-utils)
  • test-evaluator-helpers - verifies shared file-read helpers (text, CSV, Excel, ownCloud lookup)
  • test-rocketchat-load-npcs - verifies the Rocket.Chat service is connected and the seeded company data loaded, by asking the agent to list the members of the #engineering channel (the scorer fetches ground-truth members from the Rocket.Chat REST API)
  • test-rocketchat-npc-reply - verifies two-way messaging with an LLM-driven NPC colleague: the agent DMs zhang_wei, whose replies are generated on the host by tac_solver when -T npc_model=<model-id> is set; the scorer checks the captured reply in /workspace/ans.txt is substantive

The two Rocket.Chat test tasks bring up a Rocket.Chat 5.3 + MongoDB (replica set) stack matching upstream TheAgentCompany. Company users, channels, and messages are restored from the pinned db.dump archive via mongorestore; a data-population service seeds NPC personality definitions into Redis only (servers/rocketchat/npc-data-population). test-rocketchat-npc-reply exercises the host-side LLM NPC bridge (npc_solver.py); real stage-3 tasks that depend on colleague messaging will reuse the same mechanism.

Changelog

[3-B] - 2026-07-10

  • Give agent containers shm_size: "1gb" so Chromium/Playwright has enough /dev/shm. Docker’s default 64MB shared memory caused Page crashed / Target crashed browser failures when running multiple stage-3 tasks in parallel.

[2-B] - 2026-07-10

  • Replace message_limit=100 / -T max_messages with a 100-turn agent limit (-T max_turns) via react(on_continue=...), matching OpenHands max_iterations=100 in the original TheAgentCompany harness. Message limits count every tool result; the previous setting stopped agents after roughly half as many model steps as upstream.

[1-A] - 2026-05-19

  • Initial Inspect adaptation of The Agent Company benchmark. The full benchmark has 175 tasks; this implementation is rolled out in stages for reviewability.
  • Stage 1 (merged): five ownCloud-only example tasks to validate the approach.
  • Stage 2: twenty-eight additional ownCloud-only tasks (34 dataset samples including test tasks).
  • Per-task checkpoint scoring with separate score (pass/fail) and checkpoints (partial progress) metrics; optional improved scoring mode where original scorers are known to be brittle.
  • Test tasks for ownCloud connectivity, LLM judge, sandbox install permissions, and evaluator helpers. Rocket.Chat connectivity checks deferred to stage 3 (added in a later PR).

Static checks

Results of inspect-evals-lint at the commit this documentation was built from. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…