Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs

Evaluates LLM safety by presenting 939 prompts spanning five risk areas (information hazards, malicious uses, discrimination/toxicity, misinformation, human-chatbot harms) that responsible models should not directly answer. An LLM judge classifies each response into seven action categories (0-6), where category 6 indicates directly following the risky instruction. Metrics include refusal rate, safe rate (not category 6), and follow rate (category 6), broken down by risk area. Dataset was GPT-4-generated and human-filtered/annotated. An Inspect AI implementation of the original codebase publicly available at https://github.com/Libr-AI/do-not-answer.

Maintained By

@mkzung

Code

do_not_answer

Paper

https://arxiv.org/abs/2308.13387v2

Overview

⚠️ External evaluation. Code lives in an upstream repository. inspect_evals lists it for discoverability; review the upstream repo and pinned commit before running.

Source: mkzung/inspect-evals-do-not-answer@b386689

Evaluates LLM safety by presenting 939 prompts spanning five risk areas (information hazards, malicious uses, discrimination/toxicity, misinformation, human-chatbot harms) that responsible models should not directly answer. An LLM judge classifies each response into seven action categories (0-6), where category 6 indicates directly following the risky instruction. Metrics include refusal rate, safe rate (not category 6), and follow rate (category 6), broken down by risk area. Dataset was GPT-4-generated and human-filtered/annotated. An Inspect AI implementation of the original codebase publicly available at https://github.com/Libr-AI/do-not-answer.

Usage

Installation

This is an externally-maintained evaluation. Clone the upstream repository at the pinned commit and install its dependencies:

git clone https://github.com/mkzung/inspect-evals-do-not-answer
cd inspect-evals-do-not-answer
git checkout b386689ab2d469ea1ff9eb423ac048549f7542c5
uv sync

Running evaluations

CLI

uv run inspect eval src/do_not_answer/do_not_answer.py@do_not_answer --model openai/gpt-5-nano

Python

from inspect_ai import eval
from do_not_answer.do_not_answer import do_not_answer

eval(do_not_answer(), model="openai/gpt-5-nano")

View logs

uv run inspect view

More information

For the dataset, scorer, task parameters, and validation, see the upstream repo: mkzung/inspect-evals-do-not-answer.

Options

You can control a variety of options from the command line. For example:

uv run inspect eval src/do_not_answer/do_not_answer.py@do_not_answer --limit 10
uv run inspect eval src/do_not_answer/do_not_answer.py@do_not_answer --max-connections 10
uv run inspect eval src/do_not_answer/do_not_answer.py@do_not_answer --temperature 0.5

See uv run inspect eval --help for all available options.

More command-line options: Inspect docs ↗