Ragas

What Ragas is

Ragas is a free, self-hosted library for systematically evaluating large language model applications. It provides metrics for retrieval-augmented generation, agent and tool use, factual correctness, SQL, natural language comparisons, and other tasks. Users can define custom metrics and tailor test-set generation, prompts, and model configuration. Its experiments workflow lets teams run evaluations, inspect results, and iterate on application changes. Ragas can generate synthetic evaluation data, including test sets for RAG and agents, and describes online monitoring for quality in production. Documented integrations include LangChain, LlamaIndex, Haystack, Griptape, Arize Phoenix, and LangSmith, plus other frameworks and tracing services. Evaluation providers include OpenAI, Anthropic Claude, Google Gemini, Ollama, and OpenAI-compatible providers. Installation uses Python packages through pip or the GitHub main branch, with uvx also shown. Results appear in the console and are saved as CSV files in an experiments directory. Support questions go to Discord.