LM Evaluation Harness
LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks.
What LM Evaluation Harness is
LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks. It brings together more than 60 standard academic benchmarks, covering hundreds of subtasks and variants. Supported model options include Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM; it also works with hosted APIs such as OpenAI and Anthropic, plus local servers compatible with OpenAI APIs.
You can define custom prompts and metrics, configure tasks with YAML, and extend the harness with backends, filters, metrics, or aggregations from external packages or local modules. Publicly available prompts are intended to support reproducibility and comparison between papers. The core framework is free and MIT-licensed, with model backends and task dependencies available as separate optional installs. It is designed for self-hosted use, with API and Linux listed as platforms. One limitation: APIs without logits or log probabilities are restricted to generate-until tasks. Support is available through GitHub issues or the EleutherAI Discord.
Good
- More than 60 standard academic benchmarks
- Supports custom prompts, metrics, and YAML tasks
- Extensible through packages or local modules
- Free, MIT-licensed software
- Runtime options include CPU and accelerator providers
Watch out
- Backends and task dependencies may need separate installs
- Some APIs are limited to generate-until tasks
- MIT license provides software without warranty
Who it suits
It suits researchers, organizations, and model developers who need to evaluate language models across established or custom tasks. It may also fit teams seeking a self-hosted evaluation framework with multiple model and API options.
Verdict
LM Evaluation Harness offers a broad framework for repeatable model evaluation, with room for custom tasks and extensions. Check backend installation needs and API limitations against your intended evaluation setup.
Plans and prices
From LM Evaluation Harness's own pricing page, checked 4 Oct 2026.
| Plan | Price | What it gives you |
|---|---|---|
| LM Evaluation Harness | Free | MIT-licensed software · model backends and task dependencies may require separate optional installs |
The facts
Each one read from LM Evaluation Harness's own pages, with the date it was checked.
What it does
- API integrations
- The harness supports hosted APIs including OpenAI and Anthropic, plus local servers compatible with OpenAI APIs.checked 4 Oct 2026
- Benchmarks
- The project lists over 60 standard academic LLM benchmarks with hundreds of subtasks and variants.checked 4 Oct 2026
- Custom evaluation
- Users can define custom prompts and evaluation metrics, and can configure tasks with YAML.checked 4 Oct 2026
- Installation
- The base package provides the core framework while model backends are installed separately as optional extras.checked 4 Oct 2026
- Model support
- Supported model options include Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM.checked 4 Oct 2026
- Notable limitation
- For model APIs that do not provide logits or log probabilities, the documentation limits use to generate-until tasks.checked 4 Oct 2026
- Plugins
- Model backends, filters, metrics, and aggregations can be registered from external packages through entry points or loaded from a local module.checked 4 Oct 2026
- Purpose
- LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks.checked 4 Oct 2026
- Reproducibility
- The project says its use of publicly available prompts supports reproducibility and comparability between papers.checked 4 Oct 2026
- Security and license
- The repository provides the software under the MIT License, which states that it is provided “AS IS” without warranty.checked 4 Oct 2026
- Support
- The project directs users to open a GitHub issue or join the EleutherAI Discord for support.checked 4 Oct 2026
- Users
- The project says the harness serves as the backend for Hugging Face’s Open LLM Leaderboard and is used by research papers and organizations.checked 4 Oct 2026
Where it runs
- Supported platforms
- Documented runtime options include Windows ML and ONNX Runtime providers for CPU, CUDA, DirectML, NPU, ROCm, and other accelerators.checked 4 Oct 2026
Alternatives to LM Evaluation Harness
- 1
DeepEval
Free planOpen sourceWebAPILinuxmacOS8.3From $200 a month - 2
Promptfoo
Free planOpen sourceWebAPILinuxmacOS7.8Free plan - 3
Braintrust
Free planWebAPISelf-hosted7.7From $249 a month - 4
Giskard
Free planOpen sourceWebAPILinuxSelf-hosted7.6Free plan - 5
TruLens
Free planOpen sourceAPISelf-hosted7.2Free plan - 6
Arena (formerly Chatbot Arena)
Free planOpen sourceWebAPISelf-hosted7.1Free plan