SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security

“Security Question Answering” dataset to assess LLMs’ understanding and application of security principles. SecQA has “v1” and “v2” datasets of multiple-choice questions that aim to provide two levels of cybersecurity evaluation criteria. The questions were generated by GPT-4 based on the “Computer Systems Security: Planning for Success” textbook and vetted by humans.

Overview

SecQA (Security Question Answering) is a benchmark for evaluating the performance of Large Language Models (LLMs) in the domain of computer security. Utilizing multiple-choice questions generated by GPT-4 based on the “Computer Systems Security: Planning for Success” textbook, SecQA aims to assess LLMs’ understanding and application of security principles.

SecQA is organized into two versions: v1 and v2. Both are implemented here in both zero-shot and 5-shot settings. Version 1 is designed to assess foundational understanding, presenting questions that cover basic concepts and widely recognized security principles. It serves as a preliminary test to gauge LLMs’ basic comprehension and application of security knowledge. Version 2 introduces a higher level of difficulty with more complex and nuanced questions, pushing LLMs to demonstrate a more profound understanding and advanced reasoning in this domain.

Usage

Installation

Install with pip install inspect-evals, or uv sync from a checkout of this repository.

Running evaluations

uv run inspect eval inspect_evals/sec_qa_v1 --model openai/gpt-5-nano
uv run inspect eval inspect_evals/sec_qa_v1_5_shot --model openai/gpt-5-nano
uv run inspect eval inspect_evals/sec_qa_v2 --model openai/gpt-5-nano
uv run inspect eval inspect_evals/sec_qa_v2_5_shot --model openai/gpt-5-nano

To run multiple tasks simultaneously use inspect eval-set:

uv run inspect eval-set inspect_evals/sec_qa_v1 inspect_evals/sec_qa_v1_5_shot inspect_evals/sec_qa_v2 inspect_evals/sec_qa_v2_5_shot

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval, eval_set
from inspect_evals.sec_qa import sec_qa_v1, sec_qa_v1_5_shot, sec_qa_v2, sec_qa_v2_5_shot
eval(sec_qa_v1)
eval_set([sec_qa_v1, sec_qa_v1_5_shot, sec_qa_v2, sec_qa_v2_5_shot], log_dir='logs-run-42')

Drop uv run if you manage dependencies yourself. Log viewing (inspect view) and default-model setup are documented in the Inspect Evals README.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval inspect_evals/sec_qa_v1 --limit 10 --sample-shuffle
uv run inspect eval inspect_evals/sec_qa_v1_5_shot --max-connections 10
uv run inspect eval inspect_evals/sec_qa_v2 --temperature 0.5

See uv run inspect eval --help for all available options.

Parameters

sec_qa_v1, sec_qa_v1_5_shot, sec_qa_v2, sec_qa_v2_5_shot

No task parameters.

Dataset

Here is an example from the dataset:

Question: What is the purpose of implementing a Guest Wireless Network in a
corporate environment?

A) To provide unrestricted access to company resources

B) To offer a separate, secure network for visitors

C) To replace the primary corporate wireless network

D) To bypass network security protocols

Correct Answer: B

The model is tasked to choose one of the four options.

Scoring

A simple accuracy is calculated over the datapoints.

Changelog

[2-A] - 2026-02-16

• Migrate version to new scheme. See #907.

[1.0.1] - 2025-12-18

• Adds backoff policy for functions that connect to huggingface servers.