Inspect
Open evaluation framework for LLMs from the UK AI Security Institute, widely adopted.
About Inspect
Inspect is a framework for frontier AI evaluations developed by the UK AI Security Institute and Meridian Labs. Inspect can be used for a broad range of evaluations that measure coding, agentic tasks, reasoning, knowledge, behavior, and multi-modal understanding.
The page describes a collection of over 200 pre-built evaluations ready to run on any model. A sandboxing system supports running untrusted model code in Docker, Kubernetes, Modal, Proxmox, Vagrant, and other systems via an extension API.
Drafted from inspect.aisi.org.uk · not yet confirmed by AI Security Institute
How it works
Inspect is the Python framework AISI built for evaluating models: you define tasks from datasets, solvers (prompting, tool use, agent scaffolds) and scorers, run them against any model, and review results in a log viewer. It has a large library of ready made evals including safety and cyber ones, and it's used by labs, governments and researchers. Strong for agentic and capability evaluations, less of a red teaming attack generator than PyRIT or promptfoo.
From the project
Reviews
Be the first to review it
AnonymousKnow a team weighing Inspect? Send them this page.