18.3
DeepEval is a free, open-source framework for creating evaluation pipelines that test AI systems.
Free planOpen sourceWebAPILinux
Alternatives to PyRIT
27.8
Promptfoo helps developers and enterprises evaluate prompts and test AI applications for security risks.
Free planOpen sourceWebAPILinux
37.7
Braintrust is an observability platform for teams improving AI agents that run in production.
Free planWebAPISelf-hosted
- 4
Giskard
Giskard is a freemium platform for finding security and quality problems in AI agents before production use.
Free planOpen sourceWebAPILinuxSelf-hosted7.6Free plan - 5
TruLens
TruLens is a free, open-source Python SDK for tracing and evaluating AI agents, RAG applications, MCP apps, and summarization systems.
Free planOpen sourceAPISelf-hosted7.2Free plan - 6
Arena (formerly Chatbot Arena)
Arena (formerly Chatbot Arena) is a web-based platform for comparing AI model responses and building public leaderboards from votes.
Free planOpen sourceWebAPISelf-hosted7.1Free plan - 7
Maxim AI
Maxim AI is an evaluation and observability platform for simulating, assessing, and monitoring AI agents.
Free planWebAPISelf-hosted7.0From $29 a month - 8
Confident AI
Confident AI is an AI quality platform for evaluating application changes during development and monitoring AI applications in production.
Free planWebAPISelf-hosted6.9From $200 a month - 9
Galileo
Galileo is an observability, evaluation, and guardrail platform for generative AI and agentic applications.
Free planWebAPISelf-hosted6.9From $100 a month - 10
Parea AI
Parea AI is a platform for experiment tracking, observability, and human annotation for teams building LLM applications for production.
Free planWebAPISelf-hosted6.8From $150 a month - 11
HELM
HELM is a free, open-source Python framework for evaluating foundation models, including large language and multimodal models.
Free planWebAPISelf-hosted6.5Free plan - 12
garak
garak is a free, open-source command-line scanner for security weaknesses in language models and systems that use them.
Free planLinuxmacOSSelf-hostedWindows6.2Free plan - 13
SWE-bench
SWE-bench is a benchmark for evaluating language models on software issues collected from GitHub.
WebAPILinuxmacOS6.1 - 14
LiveBench
LiveBench is a free, open-source benchmark for evaluating large language models, designed to limit test contamination and use objective evaluation.
WebLinuxmacOSSelf-hosted6.0 - 15
LM Evaluation Harness
LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks.
Free planAPILinuxSelf-hosted5.9Free plan - 16
OpenCompass
OpenCompass is a free, open-source platform for evaluating large language models across datasets covering knowledge, reasoning, coding, science,…
Free planAPILinuxSelf-hosted5.9Free plan - 17
Inspect AI
Inspect AI is a free, open-source framework for evaluating large language models across coding, agent tasks, reasoning, knowledge, behavior and multimodal understanding.
Free planAPIExtension5.8Free plan - 18
RAGChecker
RAGChecker is a free, Apache-2.0 licensed framework for evaluating and diagnosing retrieval-augmented generation systems.
Free planSelf-hosted5.8Free plan - 19
Ragas
Ragas is a free, self-hosted library for systematically evaluating large language model applications.
APILinuxSelf-hosted5.7 - 20
ARES
ARES is a free, open-source framework for evaluating retrieval-augmented generation systems, including custom RAG models and configuration comparisons.
APILinuxSelf-hosted5.6 - 21
EvalPlus
EvalPlus is a free, open-source framework for evaluating large language models on code-related tasks.
APILinuxSelf-hosted5.6 - 22
AgentBench
APILinuxSelf-hosted5.5 - 23
DecodingTrust
APISelf-hosted5.4 - 24
HarmBench
APILinuxSelf-hosted5.4 - 25
WebArena
Self-hosted5.4 - 26
kube-bench
Linux4.6 - 27
OpenAI Evals
Web4.6 - 284.4