BixBench (Bioinformatics Analysis Benchmark)

BixBench evaluates LLM agents on open-ended bioinformatics data analysis tasks. Agents receive biological datasets and must produce Jupyter notebooks to analyze data and answer research questions. The eval uses a sandboxed IPython kernel environment where agents execute code interactively. Scoring is performed via an LLM judge comparing agent-generated answers against reference answers. The dataset is hosted on HuggingFace (futurehouse/BixBench) and tasks cover diverse bioinformatics analysis scenarios.

Maintained By

@jingedawang

@wwbmmm

Code

bixbench

Paper

https://arxiv.org/abs/2503.00096

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: concordia-ai/concordia_evals@e99597c

BixBench evaluates LLM agents on open-ended bioinformatics data analysis tasks. Agents receive biological datasets and must produce Jupyter notebooks to analyze data and answer research questions. The eval uses a sandboxed IPython kernel environment where agents execute code interactively. Scoring is performed via an LLM judge comparing agent-generated answers against reference answers. The dataset is hosted on HuggingFace (futurehouse/BixBench) and tasks cover diverse bioinformatics analysis scenarios.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/concordia-ai/concordia_evals
cd concordia_evals
git checkout e99597c8a5d68c85a5bbbb00020d7d1c813ad0e1
uv sync

Running evaluations

CLI

uv run inspect eval src/concordia_evals/bixbench/bixbench.py@bixbench --model openai/gpt-5-nano

Python

from inspect_ai import eval
from concordia_evals.bixbench.bixbench import bixbench

eval(bixbench(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: concordia-ai/concordia_evals.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/concordia_evals/bixbench/bixbench.py@bixbench --limit 10 --sample-shuffle
uv run inspect eval src/concordia_evals/bixbench/bixbench.py@bixbench --max-connections 10
uv run inspect eval src/concordia_evals/bixbench/bixbench.py@bixbench --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗