ChainForge

ChainForge is a free, open-source visual workspace for designing prompt experiments and evaluating language-model output. It suits developers, researchers and technically minded teams who want controlled comparisons across models, prompts and conversations. The strongest case is the local installation; the browser edition is convenient for sharing but omits important capabilities.

Overview

ChainForge treats prompt work as an experiment you can lay out visually rather than a sequence of ad hoc chat sessions. You choose prompt variables, models and settings, run them together and inspect the resulting responses. That makes it a good fit for comparative research and repeatable evaluation, but less suitable if you only need a simple prompt library or automated optimization; optimization is manual.

The project is in active development and described as an open beta. Ian Arawjo is its creator, chief developer and maintainer. Questions, comments and feature requests go through GitHub issues or a Google Form, so teams requiring formal enterprise support should set expectations accordingly.

Key features

Multiple LLMs can be queried in one experiment while prompts, models and model settings vary. Responses can be scored with code-based or LLM-based evaluators, then plotted for comparison. In practice, that helps expose whether a prompt change improves results consistently across providers instead of merely producing one appealing answer.

Chat Turn nodes extend testing to ongoing conversations. They can continue several conversations, template chat messages and pass a conversation from one model to another, which is useful for checking multi-turn behavior and provider handoffs.

Structured data workflows are supported through JSONL, XLSX and CSV imports, with inspected response data exportable to XLSX. Supported providers include OpenAI, Anthropic, Google Gemini and PaLM, Hugging Face, Azure OpenAI, Amazon Bedrock and Ollama. A local installation can add custom providers as Python completion functions through its Custom Providers settings.

The local build also supports Python evaluator nodes, API keys loaded from environment variables, and Ollama-hosted models. The web edition can create a unique share link for an experiment, but its reduced feature set makes it a poor substitute when evaluations depend on Python or local models.

Pricing

Free and open-source

free. There are no published plan limits, and the software is open source. This is an unusually direct fit for individuals and teams prepared to run their own workflows. It does not include API access as a product capability; you connect the providers whose credentials you supply.

The browser version does not retain API keys in cookies, local storage or the server, so keys must be entered each time ChainForge is loaded. Local encryption can protect settings or saved files, but forgetting the password permanently blocks access: passwords are stored as hashes and cannot be recovered.

Platforms

ChainForge is offered as self-hosted software and a web application. The installation guide supports Chrome, Firefox, Edge and Brave; the beta does not support other browsers. Choose self-hosting for environment variables, Python evaluation, custom providers and local Ollama or Dalai-hosted Alpaca and Llama models. Choose the web version for lighter experiments and link-based sharing, accepting its feature restrictions and repeated key entry.

Who it's for

It is a strong choice for prompt engineers, evaluators and developers who need side-by-side model comparisons, scored experiments, conversation testing or spreadsheet-backed datasets without a subscription fee. It also suits users who value open-source control and can manage local credentials and software.

It is not the best choice for nontechnical users seeking a polished hosted service, automatic prompt optimization, built-in API access or guaranteed support. Teams must also be comfortable managing encryption passwords and their own provider billing.

Pros and cons

Pros

• Broad comparative workflow: simultaneous model and prompt variants make controlled evaluations practical.
• Flexible scoring: code-based and LLM-based evaluators, plus plots, support both quantitative and model-assisted review.
• Conversation coverage: Chat Turn nodes test multi-turn flows and model handoffs.
• Open local extensibility: Python evaluators, custom providers, environment-loaded keys and local Ollama models go beyond the web edition.
• No subscription price: the open-source plan costs nothing.

Cons

• Web feature gap: browser users miss local capabilities and must re-enter API keys on every load.
• Manual optimization: users direct the experimentation rather than relying on automated prompt search.
• Password irreversibility: losing the local encryption password means losing access to protected flows or settings.
• Beta support model: the project is in open beta, with questions handled through GitHub or a form rather than a stated enterprise channel.

Alternatives

Vellum is the better fit when you want a hosted, freemium product with Android, iOS, desktop, extension, web and self-hosted availability; its Free plan starts with included tasks and pay-as-you-go credits, while Mighty costs $30 a month.

Promptfoo suits teams wanting a free Community plan with 10k red-team probes per month, all LLM evaluation features and provider integrations, runnable locally or self-hosted.

PromptHelm is preferable for a metered gateway: its free tier caps usage at 10,000 calls per month, 10 prompts, 20 versions per prompt and one seat, with seven-day log retention.

Agenta offers a free Hobby plan with two team members, 5,000 agent runs and 20 evaluations per month, plus a one-week trace-retention period.

Langfuse is a stronger hosted observability option, with a free Hobby tier including 50k units per month, 30 days of data access and two users; Core costs $29 a month.

Maxim AI provides a free Developer tier for up to three seats, one workspace, 10k logs per month and three-day retention.

Prompt Manage is aimed at prompt organization, with a free plan capped at 25 prompts and 10 runs per month.

PromptHub focuses on shared prompt access, offering unlimited public prompts and team members but no private prompts on its free plan.

For broader directories, see AI Prompt Management Software and AI Prompt Generators.

Verdict

Choose ChainForge if you want open-source, visual prompt and model experiments with serious scoring and conversation-testing flexibility, especially when you can self-host. Look elsewhere if you need a polished hosted workflow, automated optimization, persistent web credentials or support beyond a beta project's community channels.