Inspect AI
Inspect AI is a free, open-source framework for evaluating large language models across coding, agent tasks, reasoning, knowledge, behavior and multimodal un…
What Inspect AI is
Inspect AI is a free, open-source framework for evaluating large language models across coding, agent tasks, reasoning, knowledge, behavior and multimodal understanding. Evaluations bring together datasets, solvers and scorers, and the framework includes more than 200 pre-built evaluations. Inspect View provides a web interface for monitoring and visualizing results, while a VS Code extension supports log browsing, task execution and debugging. Inspect has built-in support for more than 20 model providers, plus local inference with Hugging Face, vLLM and SGLang. Its tools include web search, code execution, text editing, computer use and web browsing, and it supports MCP tools. Teams can integrate agents made with OpenAI Agents SDK, Pydantic AI and LangChain, as well as command-line agents such as Claude Code, Codex CLI and Gemini CLI. Untrusted model code can run in environments including Docker, Kubernetes, Modal, Proxmox and Vagrant. Inspect is self-hosted. Installation uses pip, and developing or running evaluations also requires access to a model.
Good
- Includes more than 200 pre-built evaluations.
- Supports more than 20 model providers.
- Offers a VS Code extension for logs and debugging.
- Can sandbox untrusted model code.
Watch out
- Evaluation development and runs require model access.
- The framework is self-hosted.
Who it suits
Inspect suits developers and teams evaluating language models across different task types. It is relevant for those who want to run evaluations themselves and need integrations with model providers, agents or sandbox environments.
Verdict
Inspect AI offers a broad framework for building and running model evaluations, with supporting tools for viewing results and working in VS Code. Factor in model access and self-hosting when planning its use.
Plans and prices
From Inspect AI's own pricing page, checked 4 Oct 2026.
| Plan | Price | What it gives you |
|---|---|---|
| Inspect AI | Free | Open-source framework for large language model evaluations |
The facts
Each one read from Inspect AI's own pages, with the date it was checked.
What it does
- Agent integrations
- Inspect can integrate agents built with OpenAI Agents SDK, Pydantic AI, and LangChain, as well as CLI agents such as Claude Code, Codex CLI, and Gemini CLI.checked 4 Oct 2026
- Editor integration
- The Inspect VS Code extension provides tools for browsing evaluation logs, running and debugging tasks, and configuring options.checked 4 Oct 2026
- Evaluation results
- Inspect View is a web-based tool for monitoring and visualizing evaluations.checked 4 Oct 2026
- Installation requirement
- The getting-started instructions install Inspect from PyPI with pip install inspect-ai and say evaluation development and runs also require access to a model.checked 4 Oct 2026
- Model integrations
- Inspect has built-in support for more than 20 model providers and local inference with Hugging Face, vLLM, and SGLang.checked 4 Oct 2026
- Provider examples
- Documented provider groups include OpenAI, Anthropic, Google, AWS Bedrock, Azure AI, Hugging Face, vLLM, Ollama, and SGLang.checked 4 Oct 2026
- Purpose
- Inspect is an open-source framework for evaluating large language models across coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding.checked 4 Oct 2026
- Reusable components
- Evaluations combine datasets, solvers, and scorers, and can use more than 200 pre-built evaluations.checked 4 Oct 2026
- Sandboxing
- The framework supports running untrusted model code in Docker, Kubernetes, Modal, Proxmox, Vagrant, and other systems through an extension API.checked 4 Oct 2026
- Tools and MCP
- Inspect includes standard tools for web search, code execution, text editing, computer use, and web browsing, and supports MCP tools.checked 4 Oct 2026
The company
- Founded
- 2024checked 28 Sep 2026
Alternatives to Inspect AI
- 1
DeepEval
Free planOpen sourceWebAPILinuxmacOS8.3From $200 a month - 2
Promptfoo
Free planOpen sourceWebAPILinuxmacOS7.8Free plan - 3
Braintrust
Free planWebAPISelf-hosted7.7From $249 a month - 4
Giskard
Free planOpen sourceWebAPILinuxSelf-hosted7.6Free plan - 5
TruLens
Free planOpen sourceAPISelf-hosted7.2Free plan - 6
Arena (formerly Chatbot Arena)
Free planOpen sourceWebAPISelf-hosted7.1Free plan