HELM
HELM is a free, open-source Python framework for evaluating foundation models, including large language and multimodal models.
What HELM is
HELM is a free, open-source Python framework for evaluating foundation models, including large language and multimodal models. It offers standardized datasets and benchmarks such as MMLU-Pro, GPQA, IFEval, and WildBench, and a unified interface to models from providers including OpenAI, Anthropic, and Google. Evaluations can measure efficiency, bias, and toxicity as well as accuracy. Users can inspect individual prompts and responses in a web interface and compare results on leaderboards, including Capabilities, Safety, and VHELM. HELM can be configured for private models, such as local checkpoints or model servers on private networks; inference can run locally or use an API. The framework supports reproducing published evaluation results and documents commands for running benchmarks, summarizing results, and starting a local web server. It is distributed under an Apache-2.0 license. HELM entered maintenance mode on June 1, 2026, with no new features or leaderboard evaluations planned. External API or provider changes may also break scenarios and models.
Good
- Free and open source under Apache-2.0.
- Includes standardized datasets and benchmarks.
- Measures efficiency, bias, and toxicity as well as accuracy.
- Supports private models and local inference.
- Web tools inspect prompts and compare results.
Watch out
- Entered maintenance mode on June 1, 2026.
- No new features or leaderboard evaluations are planned.
- External API changes may break scenarios or models.
- The project describes its coverage as incomplete.
Who it suits
HELM suits developers and researchers who want to evaluate foundation models with reproducible benchmarks and inspectable results. Its self-hosted setup may fit users who need to configure private models or local inference.
Verdict
HELM offers a broad, self-managed evaluation framework with benchmarks, inspection tools, and model configuration options. Its maintenance status and possible compatibility breaks are important considerations for ongoing use.
Plans and prices
From HELM's own pricing page, checked 8 Oct 2026.
| Plan | Price | What it gives you |
|---|---|---|
| HELM | Free | Open-source Python framework; no paid plans listed |
The facts
Each one read from HELM's own pages, with the date it was checked.
What it does
- Benchmarks
- It provides datasets and benchmarks in a standardized format, including MMLU-Pro, GPQA, IFEval and WildBench.checked 8 Oct 2026
- Compatibility limit
- Scenarios and models may break as external APIs change or providers deprecate models, and the maintainers recommend testing them for intended use.checked 8 Oct 2026
- Custom models
- Users can configure private models, including a checkpoint on local disk or a model server on a private network.checked 8 Oct 2026
- Evaluation coverage
- The project says it maintains leaderboards across domains such as medicine and finance, and aspects such as multilinguality and world knowledge.checked 8 Oct 2026
- Evaluation metrics
- Its metrics measure aspects beyond accuracy, including efficiency, bias and toxicity.checked 8 Oct 2026
- Inference options
- Model deployments can run local inference or send requests to an API.checked 8 Oct 2026
- Inspection tools
- HELM includes a web UI for inspecting individual prompts and responses and a web leaderboard for comparing results.checked 8 Oct 2026
- Inspection UI
- A web UI lets users inspect individual prompts and responses.checked 8 Oct 2026
- Install and run
- The project documents installation from PyPI with `pip install crfm-helm` and commands to run benchmarks, summarize results, and start a local web server.checked 8 Oct 2026
- Installation
- The project instructs users to install HELM from PyPI with `pip install crfm-helm`.checked 8 Oct 2026
- Known limitation
- The project describes HELM as intentionally incomplete and invites the community to identify gaps and contribute scenarios, metrics and models.checked 8 Oct 2026
- Leaderboards
- A web leaderboard compares results across models and benchmarks, with official leaderboards including Capabilities, Safety, and VHELM.checked 8 Oct 2026
- License
- The repository identifies HELM as open source and links to an Apache-2.0 license.checked 8 Oct 2026
- Maintenance
- HELM entered maintenance mode on June 1, 2026; no new features or leaderboard evaluations will be added.checked 8 Oct 2026
- Maintenance status
- The repository says HELM entered maintenance mode on June 1, 2026.checked 8 Oct 2026
- Metrics
- Its metrics cover aspects beyond accuracy, including efficiency, bias, and toxicity.checked 8 Oct 2026
- Model providers
- It offers a unified interface to models from providers including OpenAI, Anthropic and Google.checked 8 Oct 2026
- Purpose
- HELM is an open-source Python framework for holistic, reproducible, and transparent evaluation of foundation models, including language and multimodal models.checked 8 Oct 2026
- Reproducibility
- The framework can be used to reproduce published model evaluation results.checked 8 Oct 2026
- Support
- Users are directed to open GitHub issues for questions, which maintainers address when bandwidth is available.checked 8 Oct 2026
The company
- Founded
- 2022checked 28 Sep 2026
Alternatives to HELM
- 1
DeepEval
Free planOpen sourceWebAPILinuxmacOS8.3From $200 a month - 2
Promptfoo
Free planOpen sourceWebAPILinuxmacOS7.8Free plan - 3
Braintrust
Free planWebAPISelf-hosted7.7From $249 a month - 4
Giskard
Free planOpen sourceWebAPILinuxSelf-hosted7.6Free plan - 5
TruLens
Free planOpen sourceAPISelf-hosted7.2Free plan - 6
Arena (formerly Chatbot Arena)
Free planOpen sourceWebAPISelf-hosted7.1Free plan