PAWS: Paraphrase Adversaries from Word Scrambling

Evaluating models on the task of paraphrase detection by providing pairs of sentences that are either paraphrases or not.

Overview

PAWS is a dataset designed for evaluating models on the task of paraphrase detection by providing pairs of sentences that are either paraphrases or not, facilitating the training and benchmarking of natural language understanding systems.

Usage

First, install the inspect_ai and inspect_evals Python packages with:

pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals

Then, evaluate against one or more models with:

inspect eval inspect_evals/paws --model openai/gpt-4o

After running evaluations, you can view their logs using the inspect view command:

inspect view

If you don’t want to specify the --model each time you run an evaluation, create a .env configuration file in your working directory that defines the INSPECT_EVAL_MODEL environment variable along with your API key. For example:

INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>

Options

You can control a variety of options from the command line. For example:

inspect eval inspect_evals/paws --limit 10
inspect eval inspect_evals/paws --max-connections 10
inspect eval inspect_evals/paws --temperature 0.5

See inspect eval --help for all available options.

Dataset

Here is an example from the dataset (after processing by Inspect):

Answer Yes if the following two sentences are paraphrases. If they are not, answer No.  Do not give any other answer other than Yes or No in your response. Sentence1: These geographically correspond to the traditional Glasgow District , South , Edinburgh District , and North and Midlands districts . Sentence2: These geographically roughly correspond to the traditional districts of Glasgow District , South , Edinburgh District and North and Midlands . Yes

Scoring

A simple accuracy is calculated over the datapoints.