OpenCompass

Summary

OpenCompass is ranked #23 of 30 in AI LLM evaluation tools on Everything Xiaomi. It runs on API, Linux, Self-hosted, Windows.

Compared on AI LLM evaluation tools

Evaluation methods

objective; subjective; discriminative; generative; LLM-as-a-judgeopencompass.org.cn

Model support

Hugging Face models; API-based models; custom models; OpenAI; Anthropic; Gemini; Qwen; GLM; DeepSeekopencompass.org.cn

Safety evaluations

Yesopencompass.org.cn

Deployment

self-hostedopencompass.org.cn

Facts

Purpose

OpenCompass is a platform for evaluating large language and multimodal models.opencompass.readthedocs.io · 4 Oct 2026

Benchmarks

The project describes support for more than 100 datasets covering areas such as knowledge, reasoning, coding, science, language, long context, and safety.github.com · 4 Oct 2026

Scale

The project describes distributed evaluation and a scheme with 70+ datasets and about 400,000 questions.github.com · 4 Oct 2026

Model integrations

Documented model integrations include OpenAI, Gemini, Claude, Qwen, and other API models, as well as open-source models.github.com · 4 Oct 2026

API support

The installation guide documents API evaluation and lists dependencies for OpenAI, Anthropic, Qwen, and ByteDance Volcano Engine APIs.opencompass.readthedocs.io · 4 Oct 2026

Inference backends

The installation guide documents LMDeploy and vLLM as inference backend options.opencompass.readthedocs.io · 4 Oct 2026

Workflow and outputs

The quick start describes configuring, running inference and evaluation, and saving results as CSV and TXT files.opencompass.readthedocs.io · 4 Oct 2026

Reporting

The quick start says users can enable Lark reporting for status updates in Lark clients.opencompass.readthedocs.io · 4 Oct 2026

Extensibility

The project says users can add models and datasets and customize task division strategies and cluster management support.github.com · 4 Oct 2026

License

The GitHub repository links an Apache-2.0 license.github.com · 4 Oct 2026

Requirements

The installation guide states that OpenCompass requires PyTorch 1.13 or later.opencompass.readthedocs.io · 4 Oct 2026

Dataset limitation

The basic installation supports most fundamental datasets, while some datasets require additional dependencies or manual setup.opencompass.readthedocs.io · 4 Oct 2026

Intended users

The project describes CompassHub and CompassKit for researchers and practitioners evaluating language and vision-language models.github.com · 4 Oct 2026

Evaluation dimensions

Its documentation describes evaluations across language, knowledge, understanding, reasoning, safety, and specialized capabilities.doc.opencompass.org.cn · 8 Oct 2026

Distributed evaluation

The project says it supports task division and distributed evaluation.github.com · 8 Oct 2026

Datasets

OpenCompass publishes recommended configurations for supported evaluation datasets.doc.opencompass.org.cn · 8 Oct 2026

Safety evaluation

Its evaluation dimensions include fairness, legality, harmlessness, ethical considerations, and privacy protection.doc.opencompass.org.cn · 8 Oct 2026

Installation

The project documents installation with pip and from source, and its README recommends using conda to manage the Python environment.github.com · 8 Oct 2026

Compatibility limit

The README says APPS, TACO, and LiveCodeBench code-generation evaluations are unavailable without pyext, and recommends Python 3.10 for those datasets.github.com · 8 Oct 2026

Windows compatibility

A release note says OpenCC was made optional to restore Windows installation compatibility.github.com · 8 Oct 2026

Community support

The README links to Discord and WeChat and directs users to GitHub for reporting issues.github.com · 8 Oct 2026

Where it ranks on Everything Xiaomi

• Best AI LLM Evaluation Tools in 2026#23 of 30

Sources

• opencompass.readthedocs.io/en/doc_refactor/index.html· checked 4 Oct 2026
• github.com/open-compass/opencompass· checked 4 Oct 2026
• opencompass.readthedocs.io/en/latest/get_started/installation.html· checked 4 Oct 2026
• opencompass.readthedocs.io/en/latest/get_started/quick_start.html· checked 4 Oct 2026
• doc.opencompass.org.cn/user_guides/framework_overview.html· checked 8 Oct 2026
• github.com/open-compass/opencompass/blob/main/READ· checked 8 Oct 2026
• doc.opencompass.org.cn/dataset_statistics.html· checked 8 Oct 2026
• github.com/open-compass/opencompass/releases· checked 8 Oct 2026