BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents

A benchmark for evaluating agents’ ability to browse the web. The dataset consists of challenging questions that generally require web-access to answer correctly.

Contributed By

@AnselmC

Code

src/inspect_evals/browse_comp

Paper

https://arxiv.org/pdf/2504.12516

Overview

BrowseComp (“Browsing Competition”) is a benchmark for evaluating agents’ ability to browse the web. This eval is based on the author’s implementation and consists of questions that are very challenging to solve without access to a browser.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/browse_comp --model openai/gpt-5-nano

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval
from inspect_evals.browse_comp import browse_comp
eval(browse_comp)

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Web search

By default, BrowseComp runs without web access (with_browsing=False). Enable its web research agent with:

uv run inspect eval inspect_evals/browse_comp -T with_browsing=true --model openai/gpt-5-nano

The browsing agent uses Inspect’s web_search tool. OpenAI, Anthropic, Gemini, Grok, and Mistral models use their provider’s native search. Other models use Tavily and require TAVILY_API_KEY. Inspect checks this key when Tavily is used, before sending a search request. Search support depends on the model, and providers may charge separately for searches.

On supported Claude models, native search can use Anthropic-hosted code execution to filter search results.

Native Gemini 2.x search uses a text response for the final answer because those models cannot combine native search with a submission function. Other models retain the submission function.

Use -T web_search_providers=tavily to use Tavily for every model, or -T web_search_providers=exa to use Exa with EXA_API_KEY. To enable native search alongside an external fallback, pass a list, for example -T 'web_search_providers=["openai","anthropic","tavily"]'. This option only applies when with_browsing=True. Non-browsing runs and custom solvers ignore it.

Page access depends on the search provider. Some native providers support retrieving pages. Inspect’s Tavily adapter returns an answer with cited excerpts and cannot open a chosen URL. The agent has no browser tools for clicking or typing on pages. Results with this browsing agent should not be treated as comparable with results from the previous browsing agent or with different search providers.

Neither built-in solver requires Docker. Custom solvers that require a sandbox must set the sandbox task parameter or Inspect’s --sandbox option and provide any required sandbox configuration.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/browse_comp --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/browse_comp --max-connections 10
uv run inspect eval inspect_evals/browse_comp --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

`browse_comp`

with_browsing (bool): Use the web search agent instead of the basic solver. (default: False)
num_samples (int | None): Maximum number of dataset samples to load. (default: None)
scorer_model (str | Model | None): Model used to grade answers. Defaults to the evaluated model. (default: None)
solver (Solver | None): Custom solver. Cannot be combined with with_browsing=True. (default: None)
sandbox (str | tuple[str, str] | SandboxEnvironmentSpec | None): Optional sandbox for a custom solver that requires one. (default: None)
web_search_providers (str | list[str]): Providers for the browsing solver. Uses native search providers with Tavily fallback by default. Selecting an external provider disables native providers unless also listed, e.g. [“openai”, “tavily”]. (default: ['openai', 'anthropic', 'gemini', 'grok', 'mistral', 'tavily'])

Eval-specific options

There are several options that are specific to the BrowseComp eval and that can be set via the CLI:

with_browsing: whether to use browsing or not (default: false)
num_samples: how many samples to run (default: all samples)
scorer_model: which model to use for scoring the QA model's answer (defaults to model specified via `--model`)

Dataset

The dataset is available at https://openaipublic.blob.core.windows.net/simple-evals/browse_comp_test_set.csv. The entries in the dataset are encrypted and can be decrypted using the functions in utils.py.

Each entry consists of a specific question, a target answer, and a canary used for encryption and decryption. An example question (not from the dataset) and answer combination could be:

Question: What was the name of the 1995 film starring the actress who played the character Victoria that married the final scorer of the World Cup 1998 winner two years after the world cup win? 

Answer: La nouvelle tribu

Scoring

Scoring is done by an LLM that determines whether the produced answer matches the target answer. The score value is a binary choice between CORRECT and INCORRECT. In addition to the model answer, the model must provide a confidence score to accompany its answer. The model’s accuracy is determined over the totality of the CORRECT/INCORRECT answers. A calibration error is also calculated based on the models’ confidence values. While the authors of the original paper do not provide their calibration error formula, this evaluation uses the Expected Calibration Error (ECE), which is common for determining how well a model’s confidence predicts the correctness of its answer. See, for example, the PyTorch documentation for more details.

Evaluation Report

The original authors evaluated the BrowseComp dataset (1266 samples) on the following models: gpt-4o, gpt-4o w/ browsing, gpt-4.5, o1, and deep research.

At the current pricing as of May 21st, 2025, running the eval for the lowest performing model, gpt-4o, costs about $20.

Back-of-the-envelope calculations for o1 put the cost of this eval at around $350-400 making it prohibitively expensive for self-funding.

The following table shows the results produced by this eval vs the results from the original authors:

	Model	Accuracy (%)	Calibration Error (%)	Comment
Theirs	gpt-4o	0.6%	69%	No browsing
Ours	gpt-4o	0.9% (+0.3%)	56% (-13%)	No browsing
——–	——————	————	———————	—————————————————————————
Theirs	gpt-4o w/ browsing	1.9%	82%
Ours	gpt-4o w/ browsing	2.5% (+0.6%)	80% (-2%)	Evaluated on first 276 samples (22%). Using OpenAI’s built-in browsing tool
——–	——————	————	———————	—————————————————————————
Theirs	o1	9.9%	65%
Ours	o1	10% (+0.1%)	48% (-17%)	Only evaluated on first 100 samples
——–	——————	————	———————	—————————————————————————

Changelog

[5-C] - 2026-10-20

Replaced the browsing agent’s browser tools with web search and added web_search_providers. Native search is enabled for supported providers, with Tavily as the fallback. Browsing results are not comparable with earlier task versions.
Removed the default Docker sandbox and its bundled configuration. Custom solvers can still request a sandbox. Non-browsing remains the default; the dataset, answer format, and scoring are unchanged.

[4-B] - 2026-09-25

calibration_error counts samples with a confidence of exactly 0% in the lowest bin. They were left out of every bin while still counted in the denominator, so a correct answer given at 0% confidence, which is maximally miscalibrated, lowered the reported calibration error instead of raising it. Only runs with a correct answer at 0% confidence change: an incorrect answer at 0% is perfectly calibrated and adds nothing either way. See #2141.

[3-B] - 2026-08-21

browse_comp_accuracy returns nan rather than 0.0 when no sample carries the score key, so a run that measured nothing is no longer reported with the same number as a run that got every question wrong. calibration_error is unchanged. browse_comp_scorer sets score on every path, so no existing result changes and no re-run is needed. See #2230.

[2-B] - 2026-08-03

Pin sandbox image references to immutable tags/digests, identical to what latest resolved to at pin time, so registry pushes cannot silently change the evaluation environment (#2062). [IEBP005]

[2-A] - 2026-02-16

Migrate version to new scheme. See #907.

[1.1.0] - 2025-12-10

See Issue 709

Score structure updated for epoch compatibility

The scorer now returns a dictionary in Score.value containing both the correctness score and confidence value. This change ensures correct metric calculation when running evaluations with multiple epochs (--epochs).

Metric name change

accuracy → browse_comp_accuracy

Static checks

Results of inspect-evals-lint at the commit this documentation was built from. Rule names link to their documentation; findings link to the file at that commit. These checks describe structure and conventions, not whether the evaluation measures what it claims.

Loading lint results…