OpenCompass

What OpenCompass is

OpenCompass is a free, open-source platform for evaluating large language models across datasets covering knowledge, reasoning, coding, science, language, long context, and safety. It describes support for models from providers including OpenAI, Anthropic, Gemini, Qwen, GLM, and DeepSeek, with more than 100 datasets. Evaluation methods include zero-shot, few-shot, and chain-of-thought approaches using standard or dialogue prompt templates. It can divide tasks and run distributed evaluations, including data-parallel work across multiple GPUs. Configuration files record experiments, and the project supports real-time results reporting. Its modular system can be extended with models, datasets, task-division strategies, and cluster management systems. OpenCompass evaluates API models as well as open-source models; installation documentation lists LMDeploy and vLLM inference backends and an API installation option. A VLMEvalKit integration enables multimodal dataset loading, inference through OpenAI-compatible APIs, and evaluation using VLMEvalKit metrics. The Python package is intended for self-hosted use and carries an Apache-2.0 license. The README recommends Python 3.12, while pyext-backed code evaluation datasets require Python 3.10. Some datasets need extra dependencies, and some agent evaluations may bring many dependencies that conflict with existing environments.