CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge

Datasets containing 80, 500, 2000 and 10000 multiple-choice questions, designed to evaluate understanding across nine domains within cybersecurity

Overview

CyberMetric is a collection of four datasets containing 80, 500, 2000 and 10000 multiple-choice questions, designed to evaluate understanding across nine domains within cybersecurity: Disaster Recovery and BCP, Identity and Access Management (IAM), IoT Security, Cryptography, Wireless Security, Network Security, Cloud Security, Penetration Testing, and Compliance/Audit.

Contributed by @neilshaabi

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/cybermetric_80 --model openai/gpt-5-nano
uv run inspect eval inspect_evals/cybermetric_500 --model openai/gpt-5-nano
uv run inspect eval inspect_evals/cybermetric_2000 --model openai/gpt-5-nano
uv run inspect eval inspect_evals/cybermetric_10000 --model openai/gpt-5-nano

To run multiple tasks simultaneously use inspect eval-set:

uv run inspect eval-set inspect_evals/cybermetric_80 inspect_evals/cybermetric_500 inspect_evals/cybermetric_2000 inspect_evals/cybermetric_10000

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval, eval_set
from inspect_evals.cybermetric import cybermetric_80, cybermetric_500, cybermetric_2000, cybermetric_10000
eval(cybermetric_80)
eval_set([cybermetric_80, cybermetric_500, cybermetric_2000, cybermetric_10000], log_dir='logs-run-42')

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/cybermetric_80 --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/cybermetric_500 --max-connections 10
uv run inspect eval inspect_evals/cybermetric_2000 --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

cybermetric_80, cybermetric_500, cybermetric_2000, cybermetric_10000

No task parameters.

Dataset

Here is an example prompt from the dataset (after it has been further processed by Inspect):

Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of A,B,C,D.

In cryptography, what is the purpose of using a key-derivation function (KDF)?

A) Generate public keys, B) Authenticate digital signatures, C) Encrypt data using a password, D) KDF are algorithms used to transform a secret into crucial parameters like keys and Initialization Vectors (IVs).

The model is then tasked to pick the correct choice.

Scoring

A simple accuracy is calculated over the datapoints.

Changelog

[2-A] - 2026-02-16

• Migrate version to new scheme. See #907.