SEvenLLM: A benchmark to elicit, and improve cybersecurity incident analysis and response abilities in LLMs for Security Events.

Designed for analyzing cybersecurity incidents, which is comprised of two primary task categories: understanding and generation, with a further breakdown into 28 subcategories of tasks.

Overview

SEvenLLM is a benchmark to elicit, and improve cybersecurity incident analysis and response abilities in LLMs for Security Events.

Usage

Installation

This eval requires extra dependencies.

pip install inspect-evals[sevenllm]

From a checkout of this repository:

uv sync --extra sevenllm

Running evaluations

uv run inspect eval inspect_evals/sevenllm_mcq_zh --model openai/gpt-5-nano
uv run inspect eval inspect_evals/sevenllm_mcq_en --model openai/gpt-5-nano
uv run inspect eval inspect_evals/sevenllm_qa_zh --model openai/gpt-5-nano
uv run inspect eval inspect_evals/sevenllm_qa_en --model openai/gpt-5-nano

To run multiple tasks simultaneously use inspect eval-set:

uv run inspect eval-set inspect_evals/sevenllm_mcq_zh inspect_evals/sevenllm_mcq_en inspect_evals/sevenllm_qa_zh inspect_evals/sevenllm_qa_en

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval, eval_set
from inspect_evals.sevenllm import sevenllm_mcq_zh, sevenllm_mcq_en, sevenllm_qa_zh, sevenllm_qa_en
eval(sevenllm_mcq_zh)
eval_set([sevenllm_mcq_zh, sevenllm_mcq_en, sevenllm_qa_zh, sevenllm_qa_en], log_dir='logs-run-42')

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Parameters

sevenllm_mcq_zh, sevenllm_mcq_en, sevenllm_qa_zh, sevenllm_qa_en

No task parameters.

Dataset

The dataset described in the paper is a bilingual (English and Chinese) instruction dataset specifically created to train LLMs for cybersecurity tasks. It consists of 28 well-defined tasks, categorized into understanding tasks and generation tasks related to cybersecurity incident analysis.

1. Understanding Tasks (e.g., key entity recognition, malware feature extraction, vulnerability intelligence extraction)
2. Generation Tasks (e.g., attack strategy analysis, trend prediction, summary generation)
3. Size: benchmark contains 1,300 test samples with multiple-choice and query-answer questions.

Example

A phishing campaign targeted employees at a major company using fake login portals. The attacker utilized domain phishing[.]com and sent emails requesting password updates. This attack exploited a vulnerability in the email server to bypass spam filters.

Task: Key Entity Recognition

Instruction: Identify key entities such as attacker, victim, attack tools, and indicators of compromise from the text.

Output:

Attacker Information: Unknown (phishing campaign organizer)

Victim: Employees of a major company

Domain Name: phishing[.]com

Vulnerability Exploited: Email server spam filter bypass

Attack Tool: Fake login portals

Scoring

All evaluations involve key metrics such as Rouge-L, which measures overlap between model-generated and reference responses, and semantic similarity, which uses multilingual sentence transformers to assess the meaning alignment of outputs.

Changelog

[2-A] - 2026-02-16

• Migrate version to new scheme. See #907.

[2.0.0] - 2026-02-04

• Relax sentence_transformers==5.1.1 constraint to >= (introduced in 0ce63d).

[1.0.1] - 2025-12-18

• Adds backoff policy for functions that connect to huggingface servers.