18.3
DeepEval is a free, open-source framework for creating evaluation pipelines that test AI systems.
Free planOpen sourceWebAPILinux
Alternatives to Promptfoo
Promptfoo brings prompt evaluation and red-team testing together for developers and enterprises building AI applications. It compares outputs across multiple models in a web view, and its red-team simulations look for prompt injections, jailbreaks, data leaks, business-rule violations, and unsafe agent tool use. Community is free, includes all model providers and integrations, and can run locally or self-hosted. The trade-offs include a monthly cap of 10,000 red-team probes…
27.7
Braintrust is an observability platform for teams improving AI agents that run in production.
Free planWebAPISelf-hosted
37.6
Giskard is a freemium platform for finding security and quality problems in AI agents before production use.
Free planOpen sourceWebAPILinux
- 4
TruLens
TruLens is a free, open-source Python SDK for tracing and evaluating AI agents, RAG applications, MCP apps, and summarization systems.
Free planOpen sourceAPISelf-hosted7.2Free plan - 5
Arena (formerly Chatbot Arena)
Arena (formerly Chatbot Arena) is a web-based platform for comparing AI model responses and building public leaderboards from votes.
Free planOpen sourceWebAPISelf-hosted7.1Free plan - 6
Maxim AI
Maxim AI is an evaluation and observability platform for simulating, assessing, and monitoring AI agents.
Free planWebAPISelf-hosted7.0From $29 a month - 7
Confident AI
Confident AI is an AI quality platform for evaluating application changes during development and monitoring AI applications in production.
Free planWebAPISelf-hosted6.9From $200 a month - 8
Galileo
Galileo is an observability, evaluation, and guardrail platform for generative AI and agentic applications.
Free planWebAPISelf-hosted6.9From $100 a month - 9
Parea AI
Parea AI is a platform for experiment tracking, observability, and human annotation for teams building LLM applications for production.
Free planWebAPISelf-hosted6.8From $150 a month - 10
HELM
HELM is a free, open-source Python framework for evaluating foundation models, including large language and multimodal models.
Free planWebAPISelf-hosted6.5Free plan - 11
garak
garak is a free, open-source command-line scanner for security weaknesses in language models and systems that use them.
Free planLinuxmacOSSelf-hostedWindows6.2Free plan - 12
PyRIT
PyRIT is a free, open-source framework for security professionals and engineers evaluating risks in generative AI systems.
Free planWebAPILinuxSelf-hosted6.1Free plan - 13
SWE-bench
SWE-bench is a benchmark for evaluating language models on software issues collected from GitHub.
WebAPILinuxmacOS6.1 - 14
LiveBench
LiveBench is a free, open-source benchmark for evaluating large language models, designed to limit test contamination and use objective evaluation.
WebLinuxmacOSSelf-hosted6.0 - 15
LM Evaluation Harness
LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks.
Free planAPILinuxSelf-hosted5.9Free plan - 16
OpenCompass
OpenCompass is a free, open-source platform for evaluating large language models across datasets covering knowledge, reasoning, coding, science,…
Free planAPILinuxSelf-hosted5.9Free plan - 17
Inspect AI
Inspect AI is a free, open-source framework for evaluating large language models across coding, agent tasks, reasoning, knowledge, behavior and multimodal understanding.
Free planAPIExtension5.8Free plan - 18
RAGChecker
RAGChecker is a free, Apache-2.0 licensed framework for evaluating and diagnosing retrieval-augmented generation systems.
Free planSelf-hosted5.8Free plan - 19
Ragas
Ragas is a free, self-hosted library for systematically evaluating large language model applications.
APILinuxSelf-hosted5.7 - 20
ARES
ARES is a free, open-source framework for evaluating retrieval-augmented generation systems, including custom RAG models and configuration comparisons.
APILinuxSelf-hosted5.6 - 21
EvalPlus
EvalPlus is a free, open-source framework for evaluating large language models on code-related tasks.
APILinuxSelf-hosted5.6 - 22
AgentBench
APILinuxSelf-hosted5.5 - 23
DecodingTrust
APISelf-hosted5.4 - 24
HarmBench
APILinuxSelf-hosted5.4 - 25
WebArena
Self-hosted5.4 - 26
kube-bench
Linux4.6 - 27
OpenAI Evals
Web4.6 - 284.4