RAGChecker
RAGChecker is a free, Apache-2.0 licensed framework for evaluating and diagnosing retrieval-augmented generation systems.
What RAGChecker is
RAGChecker is a free, Apache-2.0 licensed framework for evaluating and diagnosing retrieval-augmented generation systems. It reports overall pipeline measurements and separate diagnostics for retrieval and generation. Fine-grained checks operate at the claim level through entailment operations. Retriever metrics include claim recall and context precision, while generator metrics cover context utilization, noise sensitivity, hallucination, self-knowledge, and faithfulness. Each query requires a ground-truth answer annotation; generated responses and retrieved context can also be supplied. The framework runs self-hosted through a command-line interface or Python API, with selectable metric groups or all metrics. Installation uses pip install ragchecker and requires the spaCy en_core_web_sm model in the quick start. Model choices are configurable, with an AWS Bedrock Llama 3 70B example. Documentation also covers a released benchmark and LlamaIndex integration.
Good
- Free open-source Apache-2.0 license
- Claim-level entailment evaluation
- Separate retriever and generator metrics
- CLI and Python API access
- LlamaIndex integration documented
Watch out
- Self-hosted deployment required
- Ground-truth answer annotation is required
- Quick start requires a spaCy model
Who it suits
It suits developers and teams evaluating self-hosted RAG applications that need retrieval and generation diagnostics. Users should be comfortable installing Python packages and supplying annotated evaluation data.
Verdict
RAGChecker provides a focused, self-hosted way to measure and diagnose RAG pipelines. Its claim-level metrics, configurable models, and Python or CLI use fit technical evaluation workflows.
Plans and prices
From RAGChecker's own pricing page, checked 4 Oct 2026.
| Plan | Price | What it gives you |
|---|---|---|
| RAGChecker | Free | Apache-2.0 licensed open-source framework · installable with pip |
The facts
Each one read from RAGChecker's own pages, with the date it was checked.
What it does
- Benchmark
- The repository says its benchmark was released and links a guide for using it.checked 4 Oct 2026
- Fine-grained evaluation
- Its fine-grained evaluation uses claim-level entailment operations.checked 4 Oct 2026
- Generator metrics
- Documented generator metrics include context utilization, noise sensitivity, hallucination, self-knowledge, and faithfulness.checked 4 Oct 2026
- Input requirement
- For each query, the only required annotation is the ground-truth answer; generated responses and retrieved context are also accepted in the input format.checked 4 Oct 2026
- Installation
- The maker documents installation with pip install ragchecker and requires downloading the spaCy en_core_web_sm model in its quick start.checked 4 Oct 2026
- Integration
- The README documents an integration with LlamaIndex for evaluating RAG applications built with it.checked 4 Oct 2026
- License
- The project is licensed under Apache-2.0.checked 4 Oct 2026
- Model setup
- The example pipeline uses configurable extractor and checker models and documents an AWS Bedrock Llama 3 70B example.checked 4 Oct 2026
- Pipeline metrics
- It provides overall metrics for the RAG pipeline and diagnostic metrics for retrieval and generation components.checked 4 Oct 2026
- Purpose
- RAGChecker is an automatic evaluation framework for assessing and diagnosing retrieval-augmented generation systems.checked 4 Oct 2026
- Retriever metrics
- Documented retriever metrics include claim recall and context precision.checked 4 Oct 2026
- Security reporting
- The contribution guide asks users to report potential security issues to AWS/Amazon Security through its vulnerability reporting page rather than public GitHub issues.checked 4 Oct 2026
- Support and contributions
- The project directs bug reports and feature requests to its GitHub issue tracker and welcomes pull requests.checked 4 Oct 2026
- Usage
- The framework can be run through a CLI or Python API, with selectable metric groups or all metrics.checked 4 Oct 2026
Alternatives to RAGChecker
- 1
DeepEval
Free planOpen sourceWebAPILinuxmacOS8.3From $200 a month - 2
Promptfoo
Free planOpen sourceWebAPILinuxmacOS7.8Free plan - 3
Braintrust
Free planWebAPISelf-hosted7.7From $249 a month - 4
Giskard
Free planOpen sourceWebAPILinuxSelf-hosted7.6Free plan - 5
TruLens
Free planOpen sourceAPISelf-hosted7.2Free plan - 6
Arena (formerly Chatbot Arena)
Free planOpen sourceWebAPISelf-hosted7.1Free plan