BFCL: Berkeley Function-Calling Leaderboard

Evaluates LLM function/tool-calling ability on a simplified split of the Berkeley Function-Calling Leaderboard (BFCL).

Overview

The Berkeley Function-Calling Leaderboard (BFCL) is the first comprehensive evaluation on the LLM’s ability to call functions and tools.

It consists of four fairly distinct generations V1 — single-turn function calling (AST matching). V2 — same format but user-contributed “live” samples. V3 — multi-turn with stateful backends. V4 — agentic: memory persistence across sessions and multi-hop web search.

This port of the BFCL take the official BFCL implementation and ports it to the Inspect AI framework. This implementation consists of the following components:

1. utils/ Contains the information on how to configure each category group. Each category requires a specific solver, scorer, tools, and ground truth.
2. score/ Contains the scoring functions taken from the official implementation. AST matching for single-turn (scorer.py), state+response checking for multi-turn V3 (multi_turn_scorer.py), and text-answer matching for agentic V4 (agentic_scorer.py).
3. solve/ Contains the solvers, some which require multiple turns, tool set up and execution.
4. data.py Downloads the dataset stored in the official implementation.
5. backends/ Downloads and loads the upstream stateful backend API classes (file system, trading bot, etc.) used by multi-turn and agentic categories (loader.py, downloader.py).
6. bfcl.py Coordinate the components to run the evals

High-level eval design

Each BFCL sample contains a user query that requires tool calling. The model is generally scored on whether it successfully uses the correct/ relevant tool with the right information to complete the user query. Each sample is part of a category (outlined in utils/CATEGORIES.md) which specifies how the model is prompted, scored, and what tools it is provided.

• Single-turn (V1/V2): The model receives one user message and tool schemas. It must produce the correct function call(s). Scored by AST matching against ground truth.
• Multi-turn (V3): The model works through a sequence of user messages, executing tools against stateful backends (file system, trading bot, etc.). Scored by comparing final backend state and execution results.
• Agentic (V4): The model uses tools autonomously — either chaining web searches to answer multi-hop questions, or storing and retrieving information across sessions via memory backends. Scored by checking the final text answer.

Some of the V4 categories are memory categories that require 2 steps to run. First bfcl_prereqs runs conversations where the model-under-testing stores facts using memory tools saving them to a directory. Then, running the bfcl task starts a fresh session with only the saved memory snapshot and asks questions the model must answer by querying its stored data. This tests whether the model knows what to store and how to retrieve it.

Installation

There are two ways of using Inspect Evals, from pypi as a dependency of your own project and as a standalone checked out GitHub repository.

If you are using it from pypi, install the package and its dependencies via:

pip install inspect-evals[bfcl]

If you are using Inspect Evals in its repository, install the base dependencies with:

uv sync --extra bfcl

To also run the V4 memory categories (memory_kv, memory_vector, memory_rec_sum), install the additional backend dependencies:

uv sync --extra bfcl_v4

The bfcl_v4 extra includes overrides, rank-bm25, sentence-transformers, and faiss-cpu. The memory_vector backend will download a small sentence-transformer model (~90 MB) on first use.

Execution

To run the BFCL evaluation, use the following command:

inspect eval inspect_evals/bfcl --model openai/gpt-4o-mini

Default run

The default categories list includes every V1, V2, and V3 category (except rest and the meta-index format_sensitivity) — 4981 samples with no external setup required. The V4 agentic categories are opt-in because they depend on external state:

• Memory categories (memory_kv, memory_vector, memory_rec_sum — 465 samples total) require a snapshot built by a prior bfcl_prereqs run, referenced via -T snapshot_id=<id>. Requesting them without a valid snapshot raises a ValueError listing the snapshot ids available on disk.
• Web search categories (web_search_base, web_search_no_snippet — 200 samples total) require SERPAPI_API_KEY. If the key is absent they are skipped with a warning (or, if they were the only categories requested, a ValueError).

Memory categories (V4)

Memory categories require two tasks to be run in order with the same model and the same snapshot_id. First, run bfcl_prereqs to build the memory snapshot, then run bfcl to evaluate the real questions against it:

inspect eval inspect_evals/bfcl_prereqs --model <model> -T categories=memory_kv -T snapshot_id=<id>
inspect eval inspect_evals/bfcl --model <model> -T categories=memory_kv -T snapshot_id=<id>

bfcl_prereqs accepts the same categories values as bfcl but only accepts memory categories (memory_kv, memory_vector, memory_rec_sum, or the memory alias). The shorthand alias memory expands to all three variants:

inspect eval inspect_evals/bfcl_prereqs --model <model> -T categories=memory -T snapshot_id=<id>
inspect eval inspect_evals/bfcl --model <model> -T categories=memory -T snapshot_id=<id>

Notes on the two-step workflow:

• Run bfcl_prereqs and bfcl with matching categories (e.g. both memory_kv, or both memory) — bfcl reads the snapshot for each requested memory variant, and a variant that was not prepared under the given snapshot_id starts from empty memory.
• The same model must be used for both tasks — the snapshot records that specific model’s tool calls.
• The snapshot check verifies the id exists on disk but not that the prereq run completed: an interrupted bfcl_prereqs run leaves a partial snapshot that passes the check (see Issues). Re-run bfcl_prereqs to completion under a fresh snapshot_id if a run was interrupted.

Limitations

This implementation supports all V1 (original), V2 (live), V3 (multi-turn), and V4 (agentic) categories, with the following exceptions:

• rest: Not implemented. Requires executing live HTTP requests against external RapidAPI endpoints and comparing responses. The category was retired upstream in PR #943.
• web_search_*: Not part of the default run. Requires a SERPAPI_API_KEY environment variable when requested.
• format_sensitivity: Not a standalone dataset — it is a meta-index over other categories and cannot be evaluated directly.

See also

• The official BFCL GitHub Repo
• BFCL V1 Blog Post
• BFCL V2 Blog Post
• BFCL V3 Blog Post
• BFCL V4 Web Search Blog Post
• BFCL V4 Memory Blog Post

Parameters

bfcl

• categories (str | list[str]): (default: ['exec_multiple', 'exec_parallel', 'exec_parallel_multiple', 'exec_simple', 'irrelevance', 'live_irrelevance', 'live_multiple', 'live_parallel', 'live_parallel_multiple', 'live_relevance', 'live_simple', 'multi_turn_base', 'multi_turn_composite', 'multi_turn_long_context', 'multi_turn_miss_func', 'multi_turn_miss_param', 'multiple', 'parallel', 'parallel_multiple', 'simple_java', 'simple_javascript', 'simple_python', 'sql'])
• snapshot_id (str | None): (default: None)

bfcl_prereqs

• categories (str | list[str]): (default: ['memory_kv', 'memory_rec_sum', 'memory_vector'])
• snapshot_id (str | None): (default: None)

Issues

• Implement the Rest category.
• Implement the execution scorer for the exec categories. This is low priority as it this code was never published and these categories retired as of PR #943..
• Going through the logs as there might be some false positives. A trajectory analysis of 100 samples was done and found to have valid results, but there are other smaller things to check for example Invalid value for function: 'lambda t: 3t**2 + 2t + 1'. Expected one of ['lambda x: 3x**2 + 2x + 1'] where the two answers are syntactically equivalent but marked incorrectly. Or a sample that was marked incorrect because a tweet was posted with ..."psi = 3.0" rather than the ground truth which was ..."psi =3" (argueably, the first first is more correct). A trajectory analysis of 100 samples was done on the single-turn categories which returned normal findings.
• Do an error analysis on the multi-turn categories. Based off a visual inspection of a handful of incorrect samples there maybe False negatives. For example ‘multi_turn_base_45 was marked incorrect as it didn’t do the search with the key term ’draft’ however this did not impact model output and answer which was able to identify the names with the term draft in it.
• Provide a ‘grouping’ helper function to format the results in the same way as the live leaderboard (categories are grouped in specific ways.). See https://github.com/ShishirPatil/gorilla/blob/cf12f01fc5582837cfcb496e78bc5dafd18f5f0e/berkeley-function-call-leaderboard/bfcl_eval/eval_checker/eval_runner_helper.py#L288 for the official implementation mapping eval results to their leaderboard.
• AST scorer implements one-level-deep element type checking for arrays. To be more completely, this should be recursive, but following the the official BFCL implementation, which also only checks one level deep, we did not implement recursion to avoid unnecessary complexity.
• V4 agentic scores need re-running after the loader refactor that switched to spec_from_file_location and cleaned up sys.modules pollution. The V4 results in the evaluation report predate this change.
• Memory snapshot completeness is not verified. The snapshot_id check only confirms the snapshot directory exists on disk, but the backends rewrite {scenario}_final.json after every prerequisite conversation, so an interrupted bfcl_prereqs run leaves a partial snapshot that passes the check and silently depresses memory scores. A completeness manifest (written only after the final entry of each chain, recording the model used) should replace the existence check.
• memory_vector backend may have known environment issues. The upstream memory_vector.py creates a SentenceTransformer model and imports FAISS at module level. On macOS, PyTorch and FAISS ship conflicting OpenMP runtimes that can crash (SIGSEGV) on the first FAISS parallel operation (see pytorch#149201). A previously implemented fix called faiss.omp_set_num_threads(1) (and set OMP_NUM_THREADS=1) after the MemoryAPI_vector backend loads, but it was removed because the crash was not reproducible.
• OpenAI gpt-4.1-mini fails on multi-turn samples with Optional parameters. Three upstream backend methods (find, echo in gorilla_file_system.py and get_user_tickets in ticket_api.py) have Optional[str] type annotations. When _make_recording_tool wraps these methods, functools.wraps copies __annotations__ and inspect.signature(method) carries the Optional[str] types. Inspect AI’s schema builder (parse_tool_info → get_type_hints → json_schema) then converts these to anyOf: [{type: "string"}, {type: "null"}] in the tool JSON schema. GPT-4.1-mini rejects this with a misleading “could not parse the JSON body” error (400 BadRequestError). GPT-5.1 handles the same schema correctly. Recommended fix: In _make_recording_tool (multi_turn_solver.py), after setting the __signature__, strip Optional wrappers from parameter annotations (unwrap Optional[X] / X | None to X) and patch both __signature__ and __annotations__ before the @tool decorator builds the schema. This keeps the fix local to BFCL without modifying the Inspect AI framework.

Evaluation Report

Model	Provider	Accuracy (Inspect)	Standard Error (Inspect)	Time
claude-haiku-4-5-20251001	Anthropic	0.806	0.006	11m 55s
gpt-4.1-mini-2025-04-14	OpenAI	0.788	0.006	10m 3s

Results by Category

claude-haiku-4-5-20251001

Category	Accuracy	Stderr
exec_multiple	0.860	0.050
exec_parallel	0.780	0.059
exec_parallel_multiple	0.725	0.071
exec_simple	0.940	0.024
irrelevance	0.846	0.023
live_irrelevance	0.848	0.012
live_multiple	0.767	0.013
live_parallel	0.750	0.112
live_parallel_multiple	0.708	0.095
live_relevance	0.625	0.121
live_simple	0.767	0.026
multiple	0.940	0.017
parallel	0.905	0.021
parallel_multiple	0.870	0.024
simple_java	0.380	0.049
simple_javascript	0.220	0.059
simple_python	0.928	0.013
sql	0.440	0.050

gpt-4.1-mini-2025-04-14

Category	Accuracy	Stderr
exec_multiple	0.840	0.052
exec_parallel	0.820	0.055
exec_parallel_multiple	0.700	0.073
exec_simple	0.940	0.024
irrelevance	0.850	0.023
live_irrelevance	0.782	0.014
live_multiple	0.774	0.013
live_parallel	0.750	0.112
live_parallel_multiple	0.625	0.101
live_relevance	0.812	0.098
live_simple	0.733	0.028
multiple	0.915	0.020
parallel	0.915	0.020
parallel_multiple	0.870	0.024
simple_java	0.460	0.050
simple_javascript	0.360	0.069
simple_python	0.930	0.013
sql	0.170	0.038

• Evaluation date: 2026-03-06
• Evaluation version: 3-B
• Total samples: 3,981
• --sample-shuffle randomises sample order to mitigate positional bias.
• uv run inspect eval inspect_evals/bfcl --model openai/gpt-4.1-mini-2025-04-14,anthropic/claude-haiku-4-5-20251001 --sample-shuffle

V3 Multi-Turn Results

Model	Provider	Accuracy (Inspect)	Standard Error (Inspect)	Time
claude-haiku-4-5-20251001	Anthropic	0.419	0.016	35m 49s
gpt-5.1-2025-11-13	OpenAI	0.273	0.037	7m 38s

claude-haiku-4-5-20251001

Category	Accuracy	Stderr
multi_turn_base	0.575	0.035
multi_turn_miss_func	0.445	0.035
multi_turn_miss_param	0.535	0.035
multi_turn_long_context	0.525	0.035
multi_turn_composite	0.015	0.009

gpt-5.1-2025-11-13

Category	Accuracy	Stderr
multi_turn_base (n=28)	0.464	0.096
multi_turn_miss_func (n=39)	0.308	0.075
multi_turn_miss_param (n=22)	0.273	0.097
multi_turn_long_context (n=28)	0.357	0.092
multi_turn_composite (n=33)	0.000	0.000

• Note that only 10% of the eval was ran for cost purposes (where n is the number of samples in that category).

• Evaluation date: 2026-03-24

• Evaluation version: 4-B

• claude-haiku-4-5-20251001: 1,000 samples (200 per category × 5 categories). multi_turn_base and multi_turn_long_context from initial run; multi_turn_miss_func, multi_turn_miss_param, and multi_turn_composite rerun after bug fix.

• gpt-5.1-2025-11-13: 150 samples (30 per category × 5 categories)

• --sample-shuffle randomises sample order to mitigate positional bias.

• uv run inspect eval inspect_evals/bfcl -T "categories=['multi_turn_base','multi_turn_miss_func','multi_turn_miss_param','multi_turn_long_context','multi_turn_composite']" --model anthropic/claude-haiku-4-5-20251001 --sample-shuffle

• uv run inspect eval inspect_evals/bfcl -T "categories=['multi_turn_base','multi_turn_miss_func','multi_turn_miss_param','multi_turn_long_context','multi_turn_composite']" --model openai/gpt-5.1-2025-11-13 --limit 150 --sample-shuffle

V4 Agentic Results

claude-haiku-4-5-20251001

Category	Accuracy	Stderr
web_search_base	0.790	0.041
web_search_no_snippet	0.815	0.041
memory_kv	0.452	0.040
memory_rec_sum	0.348	0.038
memory_vector	0.632	0.039

• --max-connections 1 is required for web search (avoids SERP API rate limits).
• Evaluation date: 2026-05-12
• Evaluation version: 6-B

uv run inspect eval inspect_evals/bfcl -T categories=web_search_base --model anthropic/claude-haiku-4-5-20251001 --max-connections 1
uv run inspect eval inspect_evals/bfcl -T categories=web_search_no_snippet --model anthropic/claude-haiku-4-5-20251001 --max-connections 1
uv run inspect eval inspect_evals/bfcl_prereqs -T categories=memory_kv --model anthropic/claude-haiku-4-5-20251001
uv run inspect eval inspect_evals/bfcl -T categories=memory_kv --model anthropic/claude-haiku-4-5-20251001
uv run inspect eval inspect_evals/bfcl_prereqs -T categories=memory_rec_sum --model anthropic/claude-haiku-4-5-20251001
uv run inspect eval inspect_evals/bfcl -T categories=memory_rec_sum --model anthropic/claude-haiku-4-5-20251001
uv run inspect eval inspect_evals/bfcl_prereqs -T categories=memory_vector --model anthropic/claude-haiku-4-5-20251001
uv run inspect eval inspect_evals/bfcl -T categories=memory_vector --model anthropic/claude-haiku-4-5-20251001

Results Comparison to Paper

The table compares the results of the Inspect Evals run with the BFCL leaderboard (Last Updated: 2026-03-06).

FC means they used native support for function/tool calling whilst Prompt describes the walk-around option for function calling, using model’s normal text generation capability.

claude-haiku-4-5-20251001

Category	Notes on how to calculate	Inspect Evals Run	FC	Prompt
Simple Function (AST)	Unweighted mean of simple_python, simple_java, simple_javascript	0.509	0.710	0.557
Multiple Function (AST)	multiple	0.940	0.940	0.840
Parallel Function (AST)	parallel	0.905	0.925	0.380
Parallel Multiple (AST)	parallel_multiple	0.870	0.885	0.440
Live Simple (AST)	live_simple	0.767	0.837	0.667
Live Multiple (AST)	live_multiple	0.767	0.776	0.498
Live Parallel (AST)	live_parallel	0.750	0.750	0.563
Live Parallel Multiple (AST)	live_parallel_multiple	0.708	0.750	0.167
Live Relevance	live_relevance	0.625	0.625	0.313
Live Irrelevance	live_irrelevance	0.848	0.851	0.953

gpt-4.1-mini-2025-04-14

Category	Notes on how to calculate	Inspect Evals Run	FC	Prompt
Simple Function (AST)	Unweighted mean of simple_python, simple_java, simple_javascript	0.583	0.733	0.749
Multiple Function (AST)	multiple	0.915	0.890	0.925
Parallel Function (AST)	parallel	0.915	0.910	0.875
Parallel Multiple (AST)	parallel_multiple	0.870	0.820	0.835
Live Simple (AST)	live_simple	0.733	0.671	0.806
Live Multiple (AST)	live_multiple	0.774	0.698	0.733
Live Parallel (AST)	live_parallel	0.750	0.438	0.813
Live Parallel Multiple (AST)	live_parallel_multiple	0.625	0.625	0.708
Live Relevance	live_relevance	0.812	0.813	0.875
Live Irrelevance	live_irrelevance	0.782	0.817	0.739

claude-haiku-4-5-20251001 (Multi-Turn)

Category	Notes on how to calculate	Inspect Evals Run	FC	Prompt
Overall Accuracy	Mean of 4 categories below (excl. composite)	0.520	0.536	0.018
Multi-Turn Base	multi_turn_base	0.575	0.635	0.015
Multi-Turn Miss Func	multi_turn_miss_func	0.445	0.425	0.000
Multi-Turn Miss Param	multi_turn_miss_param	0.535	0.525	0.040
Multi-Turn Long Context	multi_turn_long_context	0.525	0.560	0.015

Note: The leaderboard uses an unweighted average across language subcategories despite very different sample sizes (simple_python: 400, simple_java: 100, simple_javascript: 50) (see: https://github.com/ShishirPatil/gorilla/blob/cf12f01fc5582837cfcb496e78bc5dafd18f5f0e/berkeley-function-call-leaderboard/bfcl_eval/eval_checker/eval_runner_helper.py#L320).

Note: The leaderboard is now V4. Our Non-live and Live AST aggregate scores are lower than the leaderboard’s in part because we exclude exec categories from the non-live aggregate.

claude-haiku-4-5-20251001 (V4 Agentic)

Category	Notes on how to calculate	Inspect Evals Run	FC
Web Search Base	web_search_base accuracy	0.790	0.860
Web Search No Snippet	web_search_no_snippet accuracy	0.815	0.810
Web Search (avg)	Unweighted mean of web search categories	0.803	0.835
Memory KV	memory_kv_acc (155 task samples)	0.452	0.516
Memory Rec Sum	memory_rec_sum_acc (155 task samples)	0.348	0.561
Memory Vector	memory_vector_acc (155 task samples)	0.632	0.555
Memory (avg)	Unweighted mean of memory categories	0.477	0.544

Note: For memory categories, the leaderboard scores 155 task samples only; memory_*_acc in the Inspect run covers these same 155 samples. Leaderboard FC values are from the BFCL-Result 2025-12-16 snapshot.

Note: The leaderboard appears to exclude the non-live irrelevance score from the Hallucination Measurements (relevance and irrelevance).

Note: Multi-turn composite is excluded from the overall accuracy comparison as it is missing from the paper results (figure 1 on page 6).

Changelog

[7-B] - 2026-08-26

• Added V4 agentic categories (opt-in, not part of the default run): memory_kv, memory_vector, memory_rec_sum, web_search_base, web_search_no_snippet.
• Added agentic scorer (final text answer match) for V4 categories.
• Memory prerequisite setup is a separate bfcl_prereqs task (run before bfcl for memory categories). It produces no scored metrics — its only purpose is to build the memory snapshot on disk. Snapshots are stored under a snapshot_id passed to both tasks; bfcl raises a ValueError when memory categories are requested without a valid one.
• Web search categories are skipped with a warning when SERPAPI_API_KEY is not set.
• Multi-turn tool schemas now strip Optional[X] / X | None to X before being sent to the model. Inspect AI’s schema builder previously emitted anyOf: [type, null], which some model APIs (e.g. GPT-4.1-mini, OpenAI strict mode) reject. Backend defaults already handle the absent-value case, so this is safe. This applies to all multi-turn categories, including V3 (multi_turn_base, multi_turn_composite, multi_turn_long_context, multi_turn_miss_func, multi_turn_miss_param) and the new V4 agentic categories. Scores on strict-schema models may differ from prior versions.

[6-B] - 2026-07-29

• Fix the AST scorer penalising optional parameters at their schema default. An optional parameter left out is the same call as one passed at its declared default, so both directions are now treated as equivalent: passing the default when the ground truth omits it is no longer Unexpected parameter, and omitting it when the ground truth states the default is no longer Missing parameter. Explicit non-default values, required parameters and parameters absent from the schema are unchanged. See #2004.

[5-B] - 2026-04-01

• Fix crash in multi-turn solver when missed_function_docs is empty for a turn but missed_function_names has entries.

[4-B] - 2026-03-12

• Added multi-turn categories
• Added in a multi-turn solver and scorer which included the import of function modules

[3-B] - 2026-02-20

• Renamed the category parameter to categories.
• Expanded from single category (exec_simple) to all V1 and V2 categories(except rest).
• Updated scorers (matching functions) to match the official implementation more closely.

[2-A] - 2026-02-16

• Migrate version to new scheme. See #907.

[1.0.2] - 2026-02-02

• This update aims to preserve the core logic of the exec_simple category while updating the data source from a Huggingface repo that has not been updated to the official github source.
• Creates a CategoryConfig object handle different categories.
• Adds shuffle parameter and sorting by ID following the official implementation.
• Adds _func_doc_language_specific_pre_processing as per the official implementation (this should not change the running of exec_simple)

[1.0.1] - 2025-12-18

• Adds backoff policy for functions that connect to huggingface servers.