IFEvalCode: Controlled Code Generation
Evaluates code generation models on their ability to produce correct code while adhering to specific instruction constraints across 8 programming languages.
Overview
IFEvalCode is a benchmark that evaluates code generation models on their ability to produce correct code while adhering to specific instruction constraints. The benchmark tests models across 8 programming languages (Python, C++, Java, JavaScript, TypeScript, C#, PHP, and Shell) with 810 samples from diverse domains.
Each sample is evaluated on two independent dimensions:
- Correctness: Does the generated code pass functional tests?
- Instruction Adherence: Does the code follow specified constraints (e.g., naming conventions, structure)
Usage
Installation
This eval requires extra dependencies.
pip install inspect-evals[ifevalcode]From a checkout of this repository:
uv sync --extra ifevalcodeRunning evaluations
uv run inspect eval inspect_evals/ifevalcode --model openai/gpt-5-nanoYou can also import tasks as normal Python objects and run them from python:
from inspect_ai import eval
from inspect_evals.ifevalcode import ifevalcode
eval(ifevalcode)Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.
Options
You can control a variety of options from the command line. For example:
uv run inspect eval inspect_evals/ifevalcode --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/ifevalcode --max-connections 10
uv run inspect eval inspect_evals/ifevalcode --temperature 0.5See uv run inspect eval --help for all available options.
Parameters
ifevalcode
languages(str | list[str] | None): Optional language filter. Can be a single language string, a list of languages, or None for all languages. Supported: python, cpp, java, javascript, typescript, csharp, php, shell. (default:None)samples_per_language(int | None): Optional number of samples to use per language for balanced evaluation. If set to 25, evaluates 200 samples. (default:None)
Dataset
The IFEvalCode dataset contains 810 coding problems across 8 programming languages. Each sample includes a coding prompt with specific instruction constraints, functional correctness tests, and instruction adherence tests.
The dataset is available on HuggingFace: Multilingual-Multimodal-NLP/IfEvalCode-testset
Scoring
Each sample is evaluated on two independent dimensions:
| Metric | Description |
|---|---|
correctness |
Whether the generated code passes functional tests |
instruction |
Whether the code follows specified instruction constraints |
overall_accuracy |
Whether both correctness AND instruction adherence pass |
Per-language metrics are also available (e.g., python_correctness, java_instruction).
Evaluation Report
Results from February 2026
| Model | Samples | Overall Accuracy | Correctness | Instruction Adherence |
|---|---|---|---|---|
| gemini-3-pro-preview | 200 | 23.5% | 55.5% | 34.0% |
| gpt-5.2 | 200 | 19.0% | 45.0% | 31.0% |
| gpt-4o-mini | 560 | 8.9% | 30.4% | 22.0% |
gpt-4o-mini is tested for comparison with its results in the original paper.
Per-language breakdown (Correctness % / Instruction Adherence %):
| Model | Python | Java | C++ | C# | TS | JS | PHP | Shell |
|---|---|---|---|---|---|---|---|---|
| gemini-3-pro-preview | 64 / 64 | 52 / 16 | 60 / 0 | 32 / 8 | 56 / 40 | 64 / 40 | 72 / 36 | 44 / 68 |
| gpt-5.2 | 72 / 44 | 48 / 16 | 40 / 4 | 12 / 4 | 40 / 28 | 40 / 44 | 72 / 40 | 36 / 68 |
| gpt-4o-mini | 44 / 29 | 30 / 10 | 19 / 4 | 17 / 3 | 24 / 27 | 47 / 34 | 43 / 27 | 19 / 41 |
Original Paper Results
The original paper evaluates 40+ models on 810 English samples across 8 programming languages.
Closed-source model results from Table 3 (Correctness % / Instruction Adherence %):
| Model | Python | Java | C++ | C# | TS | JS | PHP | Shell | Avg |
|---|---|---|---|---|---|---|---|---|---|
| o1-mini | 58.1 / 32.4 | 30.4 / 12.7 | 29.0 / 9.0 | 17.5 / 2.9 | 34.0 / 25.0 | 55.0 / 28.0 | 53.0 / 31.0 | 37.0 / 47.0 | 39.3 / 23.5 |
| o3-mini | 58.1 / 32.4 | 30.4 / 12.7 | 29.0 / 9.0 | 17.5 / 2.9 | 34.0 / 25.0 | 55.0 / 28.0 | 53.0 / 31.0 | 37.0 / 47.0 | 39.3 / 23.5 |
| o4-mini | 58.1 / 32.4 | 30.4 / 12.7 | 29.0 / 9.0 | 17.5 / 2.9 | 34.0 / 25.0 | 55.0 / 28.0 | 53.0 / 31.0 | 37.0 / 47.0 | 39.3 / 23.5 |
| Claude-3.7-Sonnet | 47.6 / 31.4 | 37.3 / 7.8 | 29.0 / 6.0 | 16.5 / 3.9 | 29.0 / 28.0 | 54.0 / 34.0 | 59.0 / 27.0 | 36.0 / 43.0 | 38.5 / 22.6 |
| GPT-4.1 | 50.5 / 31.4 | 36.3 / 12.7 | 27.0 / 6.0 | 1.0 / 3.9 | 30.0 / 27.0 | 58.0 / 25.0 | 58.0 / 26.0 | 40.0 / 52.0 | 36.8 / 23.0 |
| GPT-4.1-mini | 46.7 / 34.3 | 35.3 / 7.8 | 23.0 / 4.0 | 0.0 / 2.9 | 27.0 / 23.0 | 57.0 / 30.0 | 50.0 / 29.0 | 38.0 / 53.0 | 34.6 / 23.0 |
| GPT-4o | 45.7 / 36.2 | 25.5 / 5.9 | 22.0 / 5.0 | 15.5 / 3.9 | 30.0 / 25.0 | 47.0 / 26.0 | 51.0 / 18.0 | 32.0 / 44.0 | 33.6 / 20.5 |
| Claude-3.5-Sonnet | 48.6 / 26.7 | 33.3 / 10.8 | 19.0 / 9.0 | 12.6 / 2.9 | 24.0 / 25.0 | 48.0 / 32.0 | 41.0 / 20.0 | 28.0 / 34.0 | 31.9 / 20.0 |
| GPT-4o-mini | 49.5 / 30.5 | 27.5 / 10.8 | 10.0 / 5.0 | 7.8 / 5.8 | 29.0 / 25.0 | 47.0 / 27.0 | 42.0 / 22.0 | 23.0 / 44.0 | 29.5 / 21.2 |
| grok-3-mini-fast | 9.5 / 34.3 | 29.4 / 10.8 | 21.0 / 5.0 | 1.9 / 3.9 | 28.0 / 24.0 | 53.0 / 29.0 | 33.0 / 4.0 | 14.0 / 23.0 | 28.8 / 16.8 |
| grok-3 | 25.7 / 29.5 | 18.6 / 13.7 | 2.0 / 6.0 | 0.0 / 3.9 | 4.0 / 19.0 | 44.0 / 21.0 | 43.0 / 24.0 | 16.0 / 33.0 | 19.1 / 18.8 |
Reproducibility
Evaluation version: 1.0.0
Commands:
inspect eval inspect_evals/ifevalcode --model google/gemini-3-pro-preview -T samples_per_language=25 inspect eval inspect_evals/ifevalcode --model openai/gpt-5.2 -T samples_per_language=25 inspect eval inspect_evals/ifevalcode --model openai/gpt-4o-mini -T samples_per_language=70
Changelog
[2-A] - 2026-04-21
- Fixed TypeScript correctness always scoring 0%: the
tscinvocation now passes--types node, allowing check functions that userequire()to compile correctly against the@types/nodedeclarations installed in the sandbox.