Promptfoo

Promptfoo is an AI evaluation and security-testing platform for developers and teams building AI applications. It suits people who need to compare model outputs and probe applications for risks in one workflow. Its free Community plan is substantial, though its monthly probe cap and Node.js requirement matter for evaluation.

Overview

Promptfoo brings prompt evaluation and red-team testing together, with support for hosted and local models. Founded in 2024, the San Francisco-based company says it is now part of OpenAI and remains open source. Teams can run the Community plan locally or self-host it; larger organizations can choose managed cloud or on-premise deployment.

Key features

Evaluation across models

Promptfoo evaluates prompts across multiple models and presents outputs in a web view for comparison. Support for more than 60 providers includes OpenAI, Anthropic, Google, and local models such as Ollama. That range is useful for teams comparing providers without splitting evaluation across separate tools.

Community includes all LLM evaluation features, along with custom metrics, LLM-as-a-judge, safety evaluations, human review workflows, and CI/CD integration. Listed integrations span GitHub Actions, GitLab CI, Jenkins, Azure Pipelines, CircleCI, Bitbucket Pipelines, Splunk, Burp Suite, Google Sheets, and MCP. This breadth can connect evaluation to development and security workflows, though teams should choose integrations relevant to their own stack rather than assume every connection is essential.

Red teaming

Red-team runs simulate user behavior to look for prompt injections, jailbreaks, data leaks, business-rule violations, and insecure agent tool use. Community allows 10,000 probes per month; one probe is one request to the target system. That gives teams room for recurring checks, but the quota can constrain frequent or broad testing. Enterprise offers custom red-teaming limits and continuous monitoring.

Deployment and security

Community runs locally or self-hosted. Enterprise adds managed cloud deployment, while On-Premise runs on the customer’s infrastructure and includes complete data isolation, a dedicated runner, and an assigned deployment engineer. The site displays SOC 2 and ISO 27001 certification badges. Node.js 22.22.0 or newer is required, with Node.js 24 LTS recommended; npm, npx, and Homebrew are installation options, and Homebrew supports Mac and Linux.

Pricing

Community: free. It includes 10k red-team probes per month, all LLM evaluation features, all model providers and integrations, local or self-hosted running, and community support. It is the natural starting point for individuals and teams who can work within the probe cap and do not need enterprise support or deployment.

Enterprise: custom pricing based on team size and needs. It adds custom red-teaming limits, team sharing, continuous monitoring, SSO, API access, managed cloud deployment, priority support, and SLA guarantees. It fits organizations that need shared operations and service commitments; the trade-off is that cost depends on the team's needs.

On-Premise: custom pricing. It includes Enterprise features plus deployment on the customer's infrastructure, complete data isolation, a dedicated runner, and an assigned deployment engineer. It is the relevant upgrade for organizations requiring infrastructure control and isolation, rather than a lower-cost alternative to Community.

Platforms

Promptfoo supports API, Linux, macOS, web, Windows, and self-hosted use. Its local and self-hosted options offer deployment flexibility, but the Node.js minimum is a practical prerequisite for installation.

Who it's for

Promptfoo is a strong fit for developers and AI teams that want model comparisons, safety checks, and red teaming alongside CI/CD. The Community plan makes sense for teams that can manage their own setup and stay within 10,000 monthly probes. Organizations needing team sharing, continuous monitoring, managed cloud, SLAs, or on-premise isolation should consider a custom plan. It is less suitable for teams unable to meet the runtime requirement or needing a higher red-team quota at no cost.

Pros and cons

• Pros: Broad provider support covers major hosted models and local Ollama, making cross-model evaluation practical.
• Pros: Community includes evaluation, integrations, and local or self-hosted deployment without a paid entry tier.
• Pros: Red teaming addresses risks in prompts and agent tool use, while enterprise plans add monitoring and deployment choices.
• Cons: Community red teaming is capped at 10,000 probes per month, which can limit extensive testing.
• Cons: Installation requires Node.js 22.22.0 or newer, a hurdle for users who cannot meet that runtime requirement.
• Cons: Enterprise and On-Premise use custom pricing, so buyers must scope requirements to establish cost.

Alternatives

Choose DeepEval if you want a free, Apache 2.0-licensed open-source evaluation framework with a local and CI/CD test runner. Braintrust is another freemium option, with a Starter plan offering 1 GB of processed data, 10,000 scores, 14-day retention, and unlimited users, projects, and datasets. For a free plan with explicit team and usage caps, Confident AI offers two seats, one project, five test runs per week, and 1 GB-month of trace spans.

Galileo may suit readers seeking a paid Pro plan with 50,000 traces per month, standard RBAC, advanced analytics, and dedicated Slack support; it costs $100 a month, billed yearly. Maxim AI offers a free Developer plan with up to three seats, one workspace, 10k logs per month, three-day retention, and email support. Parea AI has a free plan capped at two team members, 3k logs per month, and 10 deployed prompts.

Whisper and garak are free alternatives for Windows, macOS, and Linux. For broader comparisons, see LLM Evaluation Tools, AI Prompt Management Software, AI Red Teaming Tools, LLM Observability Tools, AI Security Testing Tools, and AI LLM Evaluation Tools.

Verdict

Promptfoo is a compelling choice for developers and AI teams that want evaluation and red-team testing across many providers, with a free plan that supports local and self-hosted work. Its main reasons to choose it are that combination and the breadth of integrations; look elsewhere if the Node.js requirement is a barrier or Community’s probe cap is too restrictive and a custom-priced plan is not a fit.