IFEvalCode: Controlled Code Generation
Evaluates code generation models on their ability to produce correct code while adhering to specific instruction constraints across 8 programming languages.
Overview
IFEvalCode is a benchmark that evaluates code generation models on their ability to produce correct code while adhering to specific instruction constraints. The benchmark tests models across 8 programming languages (Python, C++, Java, JavaScript, TypeScript, C#, PHP, and Shell) with 810 samples from diverse domains.
Each sample is evaluated on two independent dimensions:
- Correctness: Does the generated code pass functional tests?
- Instruction Adherence: Does the code follow specified constraints (e.g., naming conventions, structure)
Usage
Installation
There are two ways of using Inspect Evals, from pypi as a dependency of your own project and as a standalone checked out GitHub repository.
If you are using it from pypi, install the package and its dependencies via:
pip install inspect-evals[ifevalcode]If you are using Inspect Evals in its repository, start by installing the necessary dependencies with:
uv sync --extra ifevalcodeRunning evaluations
Now you can start evaluating models. For simplicity’s sake, this section assumes you are using Inspect Evals from the standalone repo. If that’s not the case and you are not using uv to manage dependencies in your own project, you can use the same commands with uv run dropped.
uv run inspect eval inspect_evals/ifevalcode --model openai/gpt-5-nanoYou can also import tasks as normal Python objects and run them from python:
from inspect_ai import eval
from inspect_evals.ifevalcode import ifevalcode
eval(ifevalcode)After running evaluations, you can view their logs using the inspect view command:
uv run inspect viewFor VS Code, you can also download Inspect AI extension for viewing logs.
If you don’t want to specify the --model each time you run an evaluation, create a .env configuration file in your working directory that defines the INSPECT_EVAL_MODEL environment variable along with your API key. For example:
INSPECT_EVAL_MODEL=anthropic/claude-opus-4-1-20250805
ANTHROPIC_API_KEY=<anthropic-api-key>Options
You can control a variety of options from the command line. For example:
uv run inspect eval inspect_evals/ifevalcode --limit 10
uv run inspect eval inspect_evals/ifevalcode --max-connections 10
uv run inspect eval inspect_evals/ifevalcode --temperature 0.5See uv run inspect eval --help for all available options.
Parameters
ifevalcode
languages(str | list[str] | None): Optional language filter. Can be a single language string, a list of languages, or None for all languages. Supported: python, cpp, java, javascript, typescript, csharp, php, shell. (default:None)samples_per_language(int | None): Optional number of samples to use per language for balanced evaluation. If set to 25, evaluates 200 samples. (default:None)
Dataset
The IFEvalCode dataset contains 810 coding problems across 8 programming languages. Each sample includes a coding prompt with specific instruction constraints, functional correctness tests, and instruction adherence tests.
The dataset is available on HuggingFace: Multilingual-Multimodal-NLP/IfEvalCode-testset
Scoring
Each sample is evaluated on two independent dimensions:
| Metric | Description |
|---|---|
correctness |
Whether the generated code passes functional tests |
instruction |
Whether the code follows specified instruction constraints |
overall_accuracy |
Whether both correctness AND instruction adherence pass |
Per-language metrics are also available (e.g., python_correctness, java_instruction).
Evaluation Report
Results from February 2026
| Model | Samples | Overall Accuracy | Correctness | Instruction Adherence |
|---|---|---|---|---|
| gemini-3-pro-preview | 200 | 23.5% | 55.5% | 34.0% |
| gpt-5.2 | 200 | 19.0% | 45.0% | 31.0% |
| gpt-4o-mini | 560 | 8.9% | 30.4% | 22.0% |
gpt-4o-mini is tested for comparison with its results in the original paper.
Per-language breakdown (Correctness % / Instruction Adherence %):
| Model | Python | Java | C++ | C# | TS | JS | PHP | Shell |
|---|---|---|---|---|---|---|---|---|
| gemini-3-pro-preview | 64 / 64 | 52 / 16 | 60 / 0 | 32 / 8 | 56 / 40 | 64 / 40 | 72 / 36 | 44 / 68 |
| gpt-5.2 | 72 / 44 | 48 / 16 | 40 / 4 | 12 / 4 | 40 / 28 | 40 / 44 | 72 / 40 | 36 / 68 |
| gpt-4o-mini | 44 / 29 | 30 / 10 | 19 / 4 | 17 / 3 | 24 / 27 | 47 / 34 | 43 / 27 | 19 / 41 |
Original Paper Results
The original paper evaluates 40+ models on 810 English samples across 8 programming languages.
Closed-source model results from Table 3 (Correctness % / Instruction Adherence %):
| Model | Python | Java | C++ | C# | TS | JS | PHP | Shell | Avg |
|---|---|---|---|---|---|---|---|---|---|
| o1-mini | 58.1 / 32.4 | 30.4 / 12.7 | 29.0 / 9.0 | 17.5 / 2.9 | 34.0 / 25.0 | 55.0 / 28.0 | 53.0 / 31.0 | 37.0 / 47.0 | 39.3 / 23.5 |
| o3-mini | 58.1 / 32.4 | 30.4 / 12.7 | 29.0 / 9.0 | 17.5 / 2.9 | 34.0 / 25.0 | 55.0 / 28.0 | 53.0 / 31.0 | 37.0 / 47.0 | 39.3 / 23.5 |
| o4-mini | 58.1 / 32.4 | 30.4 / 12.7 | 29.0 / 9.0 | 17.5 / 2.9 | 34.0 / 25.0 | 55.0 / 28.0 | 53.0 / 31.0 | 37.0 / 47.0 | 39.3 / 23.5 |
| Claude-3.7-Sonnet | 47.6 / 31.4 | 37.3 / 7.8 | 29.0 / 6.0 | 16.5 / 3.9 | 29.0 / 28.0 | 54.0 / 34.0 | 59.0 / 27.0 | 36.0 / 43.0 | 38.5 / 22.6 |
| GPT-4.1 | 50.5 / 31.4 | 36.3 / 12.7 | 27.0 / 6.0 | 1.0 / 3.9 | 30.0 / 27.0 | 58.0 / 25.0 | 58.0 / 26.0 | 40.0 / 52.0 | 36.8 / 23.0 |
| GPT-4.1-mini | 46.7 / 34.3 | 35.3 / 7.8 | 23.0 / 4.0 | 0.0 / 2.9 | 27.0 / 23.0 | 57.0 / 30.0 | 50.0 / 29.0 | 38.0 / 53.0 | 34.6 / 23.0 |
| GPT-4o | 45.7 / 36.2 | 25.5 / 5.9 | 22.0 / 5.0 | 15.5 / 3.9 | 30.0 / 25.0 | 47.0 / 26.0 | 51.0 / 18.0 | 32.0 / 44.0 | 33.6 / 20.5 |
| Claude-3.5-Sonnet | 48.6 / 26.7 | 33.3 / 10.8 | 19.0 / 9.0 | 12.6 / 2.9 | 24.0 / 25.0 | 48.0 / 32.0 | 41.0 / 20.0 | 28.0 / 34.0 | 31.9 / 20.0 |
| GPT-4o-mini | 49.5 / 30.5 | 27.5 / 10.8 | 10.0 / 5.0 | 7.8 / 5.8 | 29.0 / 25.0 | 47.0 / 27.0 | 42.0 / 22.0 | 23.0 / 44.0 | 29.5 / 21.2 |
| grok-3-mini-fast | 9.5 / 34.3 | 29.4 / 10.8 | 21.0 / 5.0 | 1.9 / 3.9 | 28.0 / 24.0 | 53.0 / 29.0 | 33.0 / 4.0 | 14.0 / 23.0 | 28.8 / 16.8 |
| grok-3 | 25.7 / 29.5 | 18.6 / 13.7 | 2.0 / 6.0 | 0.0 / 3.9 | 4.0 / 19.0 | 44.0 / 21.0 | 43.0 / 24.0 | 16.0 / 33.0 | 19.1 / 18.8 |
Reproducibility
Evaluation version: 1.0.0
Commands:
inspect eval inspect_evals/ifevalcode --model google/gemini-3-pro-preview -T samples_per_language=25 inspect eval inspect_evals/ifevalcode --model openai/gpt-5.2 -T samples_per_language=25 inspect eval inspect_evals/ifevalcode --model openai/gpt-4o-mini -T samples_per_language=70