4
Evidently AI
Evidently AI evaluates, tests, and monitors machine-learning models, LLMs, RAG applications, and AI agents.
Free planOpen sourceWebAPILinuxmacOS
8.3From $80 a month
Visit
5
Weights & Biases
Weights & Biases is an AI developer platform for building AI agents, applications, and models.
Free planAndroidiOSWebAPI
8.3From $60 a month
Visit
6
Langfuse
Langfuse is an open-source AI engineering platform for teams debugging, analyzing, and iterating on AI agent applications.
Free planOpen sourceWebAPILinuxSelf-hosted
8.2From $29 a month
Visit
7
LangWatch
LangWatch is an AI agent engineering platform for teams preparing agents for production.
Free planOpen sourceWebAPILinuxmacOS
8.2From €29 a month
Visit
8
LangSmith
LangSmith is a platform for building, evaluating, deploying, monitoring, and improving AI agents.
Free planWebAPILinuxmacOS
8.0$39 usage-based
Visit
9
Promptfoo
Promptfoo helps developers and enterprises evaluate prompts and test AI applications for security risks.
Free planOpen sourceWebAPILinuxmacOS
7.8Free plan
Visit
10
Braintrust
Braintrust is an observability platform for teams improving AI agents that run in production.
Free planWebAPISelf-hosted
7.7From $249 a month
Visit
11
Giskard
Giskard is a freemium platform for finding security and quality problems in AI agents before production use.
Free planOpen sourceWebAPILinuxSelf-hosted
7.6Free plan
Visit
12
TruLens
TruLens is a free, open-source Python SDK for tracing and evaluating AI agents, RAG applications, MCP apps, and summarization systems.
Free planOpen sourceAPISelf-hosted
7.2Free plan
Visit
13
Maxim AI
Maxim AI is an evaluation and observability platform for simulating, assessing, and monitoring AI agents.
Free planWebAPISelf-hosted
7.0From $29 a month
Visit
14
Confident AI
Confident AI is an AI quality platform for evaluating application changes during development and monitoring AI applications in production.
Free planWebAPISelf-hosted
6.9From $200 a month
Visit
15
Galileo
Galileo is an observability, evaluation, and guardrail platform for generative AI and agentic applications.
Free planWebAPISelf-hosted
6.9From $100 a month
Visit
16
HELM
HELM is a free, open-source Python framework for evaluating foundation models, including large language and multimodal models.
Free planWebAPISelf-hosted
6.5Free plan
Visit
17
HoneyHive
HoneyHive is an agent observability platform for teams that want to observe, evaluate, and improve AI agents in production.
Free planWebAPISelf-hosted
6.4Free plan
Visit
18
Arize Phoenix
Arize Phoenix helps teams trace, evaluate, debug, and improve AI applications.
Free planWebAPISelf-hosted
6.3Free plan
Visit
19
Rhesis AI
Free planWebAPILinuxSelf-hosted
6.2Free plan
Visit
20
NVIDIA NeMo Evaluator
NVIDIA NeMo Evaluator is an open-source service for assessing generative AI applications, including language models, RAG pipelines, and agents.
Free planAPILinuxSelf-hosted
6.0Free plan
Visit
21
LM Evaluation Harness
LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks.
Free planAPILinuxSelf-hosted
5.9Free plan
Visit
22
Inspect AI
Inspect AI is a free, open-source framework for evaluating large language models across coding, agent tasks, reasoning, knowledge, behavior and multimodal understanding.
Free planAPIExtension
5.8Free plan
Visit
23
RAGChecker
RAGChecker is a free, Apache-2.0 licensed framework for evaluating and diagnosing retrieval-augmented generation systems.
Free planSelf-hosted
5.8Free plan
Visit
24
Ragas
Ragas is a free, self-hosted library for systematically evaluating large language model applications.
APILinuxSelf-hosted
5.7
Visit
25
UpTrain
WebAPISelf-hosted
5.7
Visit
26
ARES
ARES is a free, open-source framework for evaluating retrieval-augmented generation systems, including custom RAG models and configuration comparisons.
APILinuxSelf-hosted
5.6
Visit
27
OpenAI Evals
OpenAI Evals is an open-source platform for evaluating model outputs against style and content criteria you define.
WebAPISelf-hosted
5.6
Visit
28
OpenCompass
OpenCompass is a free, open-source platform for evaluating large language and multimodal models.
APILinuxSelf-hostedWindows
5.6
Visit
29
Pydantic Evals
Linux
5.4
Visit

Back to Lighteval