AgentClash
AgentClash is an open-source platform for evaluating AI agents on real tasks in a sandbox.
What AgentClash is
AgentClash is an open-source platform for evaluating AI agents on real tasks in a sandbox. It scores tool choices, cost, latency, recovery, and final results across multi-turn runs, and can preserve a failed run as a regression test for later evaluations. Agents run in fresh Firecracker microVMs with isolated filesystems and networks, and each sandbox is removed after a run. Challenge packs defined in YAML can specify tools, policy, scoring, and starting state; available tools include file operations, data queries, HTTP, shell, and test runners. Scoring combines deterministic, mathematical, behavioral, and language-model judges with configurable weights and consensus. Provider adapters include OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter. CI/CD checks can run through GitHub Actions, a webhook, or the CLI and fail builds when correctness, cost, latency, or required evidence regresses. AgentClash is MIT licensed and can be self-hosted as a full stack or used with its hosted backend.
Good
- Failed runs can become regression tests
- Runs agents in isolated, disposable microVMs
- Supports multiple model providers
- CI checks can gate builds on regressions
- MIT-licensed and available for self-hosting
Watch out
- Free plan allows 25 evaluation runs monthly
- Free plan retains replays for seven days
- Free plan requires a BYO LLM API key
- Enterprise pricing is custom
Who it suits
AgentClash suits teams evaluating coding, research, SRE, multi-step operations, codebase question answering, or support agents. It may fit teams that want to self-host an open-source evaluation platform or use a hosted backend.
Verdict
AgentClash links multi-turn agent evaluations with replay and regression checks, including CI/CD use. The free plan has run, model, and retention limits; paid plans start at $39 per month with annual billing.
Plans and prices
From AgentClash's own pricing page, checked 1 Oct 2026.
| Plan | Price | What it gives you |
|---|---|---|
| Free | Free | 1 workspace · 25 eval runs / month · up to 4 models per run · 7-day replay retention · BYO LLM API key · BYO E2B sandbox token · community support |
| Pro | $49 a month | 500 eval runs / workspace / month · up to 8 models per run · 30-day replay retention · hosted sandbox with included credit · private challenge packs · CI integration · 3 concurrent eval runs · email support < 1 business day · Billed monthly; annual billing $39 / month ($468 / yr) |
| Team | $100 a month | 2,000 eval runs / workspace / month · up to 12 models per run · 90-day replay retention · 10 concurrent eval runs · multiple workspaces · workspace-level audit log · Slack notifications · priority email support < 4 business hours · Billed monthly; annual billing $80 / month ($960 / yr) |
| Enterprise | Priced on request | SSO / SAML · org-wide audit logs · unlimited replay retention · 99.9% uptime SLA · dedicated support channel · custom MSA / billing terms · Custom |
The facts
Each one read from AgentClash's own pages, with the date it was checked.
What it does
- Agent evaluation
- It evaluates multi-turn agents that take actions in a real sandbox and scores tool choices, cost, latency, recovery, and the final result.checked 1 Oct 2026
- Documentation
- The public documentation covers the CLI, local stack, Fleet eval sets, datasets, regression gates, multi-turn human takeover, security stress harnesses, and runtime components.checked 1 Oct 2026
- Integrations
- CI/CD integrations can run regression tests from GitHub Actions, a webhook, or the CLI and fail builds when correctness, cost, latency, or required evidence regresses.checked 1 Oct 2026
- Knowledge sources
- Knowledge sources include PDFs, wikis, Notion, codebases, and custom APIs, with provenance attached to retrieved facts.checked 1 Oct 2026
- Open source and hosting
- AgentClash is MIT licensed, can be self-hosted as a full stack, or used against the hosted backend; its CLI installs from npm as the agentclash package.checked 1 Oct 2026
- Providers
- First-class adapters support OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter, with more than 300 models available through OpenRouter.checked 1 Oct 2026
- Purpose
- AgentClash is an open-source AI-agent evaluation platform that runs agents on real tasks, scores outcomes, replays steps, and turns failures into regression tests.checked 1 Oct 2026
- Regression loop
- When a model fails a challenge, AgentClash freezes the failing trace into a permanent test that future evaluations replay.checked 1 Oct 2026
- Sandboxing
- Each agent runs in a fresh Firecracker microVM with an isolated filesystem and network, and the sandbox is torn down after the run.checked 1 Oct 2026
- Scoring
- Runs combine deterministic, mathematical, behavioural, and LLM-based judges with configurable consensus aggregation and weights.checked 1 Oct 2026
- Security
- API keys, database credentials, and OAuth tokens are stored in a scoped secret vault and injected at tool-call time without appearing in prompts, traces, or replays.checked 1 Oct 2026
- Tools
- Agents can use file I/O, data queries, HTTP, shell, and test runners, with declarative YAML challenge packs defining tools, policy, scoring, and starting state.checked 1 Oct 2026
- Workloads
- The product is positioned for coding, research, SRE, multi-step operations, codebase question answering, and support workloads.checked 1 Oct 2026
Alternatives to AgentClash
- 1
W&B Weave
Free planOpen sourceiOSWebAPILinux8.7From $60 a month - 2
Opik
Free planOpen sourceWebAPILinuxmacOS8.4From $19 a month - 3
DeepEval
Free planOpen sourceWebAPILinuxmacOS8.3From $200 a month - 4
Future AGI AI Evaluation SDK
Free planOpen sourceWebAPILinuxmacOS8.2From $250 a month - 5
LangWatch
Free planOpen sourceWebAPILinuxmacOS8.2From €29 a month - 6
LangSmith
Free planWebAPILinuxmacOS8.0$39 usage-based