SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models

The Scientific Knowledge Evaluation benchmark is inspired by the profound principles outlined in the “Doctrine of the Mean” from ancient Chinese philosophy. This benchmark is designed to assess LLMs based on their proficiency in Studying Extensively, Enquiring Earnestly, Thinking Profoundly, Discerning Clearly, and Practicing Assiduously. Each of these dimensions offers a unique perspective on evaluating the capabilities of LLMs in handling scientific knowledge.

Contributed By

@Esther-Guo

Code

src/inspect_evals/sciknoweval

Paper

https://arxiv.org/abs/2406.09098v2

Overview

The Scientific Knowledge Evaluation (SciKnowEval) benchmark for Large Language Models (LLMs) is inspired by the profound principles outlined in the “Doctrine of the Mean” from ancient Chinese philosophy. This benchmark is designed to assess LLMs based on their proficiency in Studying Extensively, Enquiring Earnestly, Thinking Profoundly, Discerning Clearly, and Practicing Assiduously. Each of these dimensions offers a unique perspective on evaluating the capabilities of LLMs in handling scientific knowledge.

Note that this eval depends on gensim, which as of September 2025 is not compatible with python 3.13.

Usage

Install the eval-specific packages:

uv sync --extra sciknoweval
  • domain can be one of the four values: biology, chemistry, material, physics.
  • level can be one of the five values: l1, l2, l3, l4, l5.
  • task can be find in the Task Mapping session.

This will run all the tasks under L4 in Biology domain.

uv run inspect eval inspect_evals/sciknoweval -T domain=biology -T level=l4

This will run the task chemical_protocol_procedure_design in Chemistry domain.

uv run inspect eval inspect_evals/sciknoweval -T domain=chemistry -T task=chemical_protocol_procedure_design

This will run all the tasks in Material domain.

uv run inspect eval inspect_evals/sciknoweval -T domain=material

For model-graded tasks, you also need to specify the grader parameter, e.g. --model-role grader=openai/deepseek-v3-250324. By default, gpt-4.1-2025-04-14 is used.

Parameters

sciknoweval

  • domain (str | None): Filter to a single domain (biology, chemistry, material, physics). (default: None)
  • level (str | None): Filter to a single level of tasks (mutually exclusive with task). (default: None)
  • task (str | None): Filter to a single task (mutually exclusive with level). (default: None)
  • shuffle (bool): Whether to shuffle the dataset. (default: True)
  • scorer_config (dict[str, str] | None): Overrides for the default mapping of task metrics type to scorer (e.g. {“GEN_TF”: “match”}). See DEFAULT_SCORER_CONFIG in config.py for the defaults. Values may name a built-in scorer (see SCORER_REGISTRY in scorer.py), a registered scorer from another package (e.g. “my_package/my_scorer”), or a scorer in a local file (e.g. “my_scorers.py@my_scorer”). (default: None)

Environment Variables Example

OPENAI_API_KEY=xxx
INSPECT_EVAL_MODEL=openai/gpt-4.1-2025-04-14

Dataset

The dataset has been updated after the implementation, however the paper is not updated accordingly. So for now we keep the task mapping in sync with the old dataset version (v1). The newer v2 is a downsampled subset of v1 (~28k samples vs ~70k) that drops 13 tasks — including several graded by the specialized scorers, such as protein design (Smith-Waterman) and molecule generation (SMILES) — and heavily downsamples most of the tasks it keeps; adopting it would break result comparability, so the eval pins the v1 revision.

Note: Due to the dataset’s size, loading all samples and starting the eval may take 1-2 minutes when running on the entire dataset. During this time inspect’s terminal UI may also load slowly and be unresponsive.

Task Mapping

This is for referencing task names when using task parameter in the command.

Domain Level Task
Biology L1 biological_literature_qa
protein_property_identification
protein_captioning
L2 drug_drug_relation_extraction
biomedical_judgment_and_interpretation
compound_disease_relation_extraction
gene_disease_relation_extraction
detailed_understanding
text_summary
hypothesis_verification
reasoning_and_interpretation
L3 solubility_prediction
beta_lactamase_activity_prediction
fluorescence_prediction
gb1_fitness_prediction
stability_prediction
protein_protein_interaction
biological_calculation
L4 biological_harmful_qa
proteotoxicity_prediction
biological_laboratory_safety_test
L5 biological_protocol_procedure_design
biological_protocol_reagent_design
protein_design
single_cell_analysis
Chemistry L1 molecular_name_conversion
molecular_property_identification
chemical_literature_qa
molecular_captioning
L2 reaction_mechanism_inference
compound_identification_and_properties
doping_extraction
detailed_understanding
text_summary
hypothesis_verification
reasoning_and_interpretation
L3 molar_weight_calculation
molecular_property_calculation
molecular_structure_prediction
reaction_prediction
retrosynthesis
balancing_chemical_equation
chemical_calculation
L4 chemical_harmful_qa
molecular_toxicity_prediction
chemical_laboratory_safety_test
L5 molecular_generation
chemical_protocol_procedure_design
chemical_protocol_reagent_design
Material L1 material_literature_qa
L2 chemical_composition_extraction
digital_data_extraction
detailed_understanding
text_summary
hypothesis_verification
reasoning_and_interpretation
L3 valence_electron_difference_calculation
material_calculation
lattice_volume_calculation
perovskite_stability_prediction
diffusion_rate_analysis
L4 material_safety_qa
material_toxicity_prediction
L5 properties_utilization_analysis
crystal_structure_and_composition_analysis
specified_band_gap_material_generation
Physics L1 physics_literature_qa
fundamental_physics_exam
L2 detailed_understanding
text_summary
hypothesis_verification
reasoning_and_interpretation
L3 high_school_physics_calculation
general_physics_calculation
physics_formula_derivation
L4 physics_safety_qa
laboratory_safety_test
L5 physics_problem_solving

Scoring

Since there are various types of tasks in the dataset (e.g. True or False, Multiple Choice, Relation Extraction), we use different scoring methods. Each task’s metrics field in task_mapping.py selects the scorer that grades its samples:

metrics type Default scorer Method
CLS match Exact answer match (inspect built-in)
GEN_filling model_graded_fact Model-graded fact comparison (inspect built-in)
GEN_TF model_graded_tf Model-graded true/false, using the task’s evaluator prompt
GEN_MCQ model_graded_mcq Model-graded multiple choice, using the task’s evaluator prompt
GEN_SCORE model_graded_score Model-graded rating, using the task’s evaluator prompt
GEN_BLEU_ROUGE nltk_scorer Equal-weighted average of BLEU-2, BLEU-4, ROUGE-1, ROUGE-2 and ROUGE-L
GEN_MOLECULE smiles_scorer SMILES molecule similarity metrics
GEN_SMITH_WATERMAN smith_waterman_scorer Normalized Smith-Waterman sequence alignment
RE_triplets re_triplets_scorer Relation extraction; scored batch-wise by the re_f1_score metric
RE_tuples re_tuples_scorer Relation extraction; scored batch-wise by the re_f1_score metric

The default mapping lives in DEFAULT_SCORER_CONFIG (config.py); results are reported per task by the metrics_by_task metric.

For tasks with metrics value GEN_BLEU_ROUGE and GEN_MOLECULE, the implementation in Inspect is taking an average of result scores, which are directly returned in a dict in the original code.

Substituting scorers

The scorer_config task parameter overrides which scorer handles a given metrics type. Entries you don’t override keep their defaults. Values may take three forms:

  1. A built-in scorer name from the table above (see SCORER_REGISTRY in scorer.py), e.g. grade true/false tasks by exact match instead of a model grader:

    uv run inspect eval inspect_evals/sciknoweval -T scorer_config='{"GEN_TF": "match"}'

  2. A scorer in a local Python file, referenced as file.py@function_name (the same convention as inspect score --scorer). The function must be decorated with @scorer and have a -> Scorer return annotation:

    uv run inspect eval inspect_evals/sciknoweval -T scorer_config='{"CLS": "my_scorers.py@my_scorer"}'

  3. A registered scorer from an installed package, referenced by its namespaced registry name:

    uv run inspect eval inspect_evals/sciknoweval -T scorer_config='{"CLS": "my_package/my_scorer"}'

Unresolvable names or file references fail at task construction (before any samples run) with the list of valid options.

Notes for custom scorers:

  • Custom scorer factories are constructed with no arguments. Only the built-in model_graded_tf / model_graded_mcq / model_graded_score scorers receive the per-task evaluator prompts from evaluator_prompt.yaml.
  • Scores still aggregate through metrics_by_task, which applies accuracy() per task, so custom scorers should return numeric or CORRECT/INCORRECT values.
  • Substituting the RE_triplets / RE_tuples scorers is not recommended: their per-sample scores are placeholders, and the real result is computed batch-wise by the re_f1_score metric from metadata those scorers emit.

Changelog

[3-B] - 2026-07-24

  • Add a scorer_config task parameter that overrides which scorer handles each task metrics type (see Substituting scorers). Values may name a built-in scorer, a registered scorer from another package (my_package/my_scorer), or a scorer in a local file (my_scorers.py@my_scorer). Default scoring behavior is unchanged.

[3-A] - 2026-07-02

  • Fix relation-extraction scoring: metrics_by_task grouped scores under a domain-prefixed key (e.g. biology.drug_drug_relation_extraction) but checked it against the bare-name RE_TASKS list, so the check never matched and all three relation-extraction subtasks fell through to accuracy() over a placeholder Score(0) — scoring 0.0 even for correct answers. The membership check now strips the domain prefix, so re_f1_score is applied as intended. (@Le0nX)

[2-A] - 2026-02-16

  • Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

  • Adds backoff policy for functions that connect to huggingface servers.