VS Code Extension

The Inspect VS Code Extension provides a variety of tools, including:

Installation

To install, search for “Inspect AI” in the extensions marketplace panel within VS Code.

The VS Code Extension Marketplace panel is active with the search string 'Inspect AI'. The Inspect extension is selected and an overview of it appears at right.

The Inspect extension will automatically bind to the Python interpreter associated with the current workspace, so you should be sure that the inspect-ai package is installed within that environment. Use the Python: Select Interpreter command to associate a version of Python with your workspace.

Running Tasks

There are many ways to run evaluation tasks within VS Code, including:

  1. Calling inspect eval in the terminal
  2. Calling eval() directly in a notebook or script
  3. Using the Run Task button in the editor
  4. Using the Cmd+Shift+U keyboard shortcut (which will run the task currently selected in the active editor).

Two eval tasks (arc-easy and arc-challenge) in an editor, with Run Task and Debug Task buttons above them.

No matter how you run a task, when the task is complete the eval log will automatically display to the right of the active editor:

Inspect running inside Visual Studio Code. The editor shows the ARC evaluation and the log viewer at right shows results from the evaluation.

If you’d rather have the log viewer display in its own window (e.g. to move it to a different monitor) you can drag the Inspect View tab out of VS Code and move it to wherever is most convenient.

If you prefer not to have logs shown automatically, there is a setting which can disable this behaviour (search settings for “inspect log”). In that case, you can still view eval logs by clicking on the link at the bottom of the eval results:

The Inspect task results displayed in the terminal. A link to the evaluation log is at the bottom of the results display.

See the article on the Log Viewer for additional details on using it to explore eval results.

Debugging Tasks

There are two ways to debug tasks within VS Code:

  1. Using the Debug Task button in the editor

  2. Using the Cmd+Shift+T keyboard shortcut (which will debug the task currently selected in the active editor).

Two eval tasks (arc-easy and arc-challenge) in an editor, with Run Task and Debug Task buttons above them.

Note that when debugging a task, the Insepct extension will automatically limit the eval to a single sample (--limit 1 on the command line). If you prefer to debug with many samples, there is a setting that can disable the default behavior (search settings for “inspect debug”).

Activity Bar

The Inspect Activity Bar provides an interface for tuning both global configuration and task CLI arguments. Access the Activity Bar by clicking the Inspect icon on the left side of the VS Code workspace:

Inspect Activity Bar with user interface for tuning global configuration and task CLI arguments.

The activity bar has three panels:

  • Configuration edits global configuration by reading and writing values from the workspace .env config file (see the documentation on Configuration for more details on .env files).

  • Task provides a way to tweak the CLI arguments passed to inspect eval when it is run from the user interface.

  • Tasks (not shown above) lists all tasks in the current workspace, and can be used to both navigate among tasks as well as run and debug tasks directly.

Troubleshooting

If the Inspect extension is not loading into the workspace, you should investigate what version of Python it is discovering as well as whether the inspect-ai package is detected within that Python environment. Use the Output panel (at the bottom of VS Code in the same panel as the Terminal) and select the Inspect output channel using the picker on the right side of the panel:

Inspect output channel, showing the versions of Python and Inspect discovered by the extension.

Note that the Inspect extension will automatically bind to the Python interpreter associated with the current workspace, so you should be sure that the inspect-ai package is installed within that environment. Use the Python: Select Interpreter command to associate a version of Python with your workspace.