TensorZero

TensorZero is a self-hosted, open-source stack that combines an LLM gateway with observability, evaluation, optimization, and experimentation. It suits technically capable developers and teams that need provider routing and direct database control. The broad feature set costs nothing, but the archived, unmaintained project is a serious reason to avoid it for systems requiring dependable upkeep.

Overview

TensorZero places a unified API and web UI in front of major model providers, including Anthropic, AWS Bedrock, Azure, Google AI Studio, OpenAI, OpenRouter, vLLM, and xAI, while also accepting OpenAI-compatible endpoints. Routing, retries, fallbacks, prompt templates, schemas, and budget controls cover the operational basics of a multi-provider deployment. The gateway reports sub-millisecond P99 latency overhead under extreme load, a useful target for latency-sensitive services but not a substitute for current maintenance.

Inference and feedback records are written to a database you configure. Teams can query that data directly with SQL for observability, optimization, and experiments, retaining control of application data rather than handing it to a hosted service. That control comes with responsibility for deployment, database operations, security, and upgrades. The official site says TensorZero is no longer maintained, and its GitHub repository is archived and read-only.

Key features

Routing and workflows

Fallback routing and retries help keep requests moving when a provider or model fails. Traffic routing supports A/B tests, and multi-step workflows let teams coordinate more than one model call. Feedback can be attached to an individual inference or an entire episode, making comparisons more useful when a workflow has several steps. Virtual API keys and budget controls add practical ways to separate access and spending.

Data, integrations, and security

The gateway's database-centered design makes SQL analysis possible alongside the stored inference and feedback history. Documented integrations include the providers above, Together AI, OpenTelemetry, the OpenAI SDK, and clients for major programming languages. API-key authentication can protect gateway endpoints other than status and health checks. TensorZero also gathers pseudonymous aggregate usage analytics, says they exclude application data and inference inputs or outputs, and provides a setting to disable collection.

Deployment

Docker, Docker Compose, Kubernetes with Helm, and source builds are supported. This flexibility fits infrastructure teams already operating containers or clusters; it is excessive for users seeking a managed gateway with minimal administration. Questions can be directed to Slack or Discord, with team Slack or Teams access available by email.

Pricing

TensorZero LLMOps platform — free. The plan is 100% self-hosted and open source, with fallback routing, budget controls, and virtual API keys. Apache 2.0 licensing permits teams to run and adapt the software without a subscription. There are no hosted seats, service quotas, or trial-renewal terms to weigh, but the trade-off is that infrastructure, database costs, and maintenance are yours. The project described a future managed service and had no revenue model at the time covered; the current maintenance status makes that distinction especially important.

Platforms

TensorZero is offered through an API and web interface and runs on Linux in self-hosted deployments. Docker, Docker Compose, Kubernetes with Helm, or a source build provide the deployment paths.

Who it's for

Choose TensorZero if you have engineers comfortable operating a gateway and database, need one API across several LLM providers, and value SQL-level ownership of inference data. Its experimentation, workflow, and feedback model is well suited to teams comparing prompts or providers inside their own infrastructure.

Look elsewhere if you need vendor-backed updates, a managed control plane, or confidence that security and compatibility fixes will continue. An archived, read-only repository turns every future integration and operational issue into an internal maintenance task.

Pros and cons

• Pros: Free Apache 2.0 software avoids recurring license fees.
• Pros: Unified provider access, fallbacks, retries, and A/B routing support practical multi-model operations.
• Pros: User-selected database storage and direct SQL queries provide strong data control.
• Pros: Docker, Kubernetes, and source deployment cover common self-hosting environments.
• Cons: The project is no longer maintained and its GitHub repository is archived read-only.
• Cons: Self-hosting leaves deployment, database care, upgrades, and support largely with your team.
• Cons: Usage analytics are collected by default, although they can be disabled.

Alternatives

Bifrost is a freemium alternative with a free Apache 2.0 OSS plan and broader desktop platform coverage; choose it when an actively positioned option beyond TensorZero's archived project is preferable.

Routerly offers a free self-hosted plan with no markup on API calls, making it a straightforward choice when cost transparency matters more than TensorZero's experimentation stack.

YoloRouter provides a free self-hosted single binary with SQLite out of the box and optional PostgreSQL, useful when a simpler deployment is the priority.

Kong Gateway suits teams wanting a commercial gateway path, including a 30-day free trial with no gateway limits and analytics retention.

Envoy AI Gateway is a free, self-managed open-source project for teams centered on local CLI or Kubernetes deployments.

SGLang is a free open-source inference framework for users who need model serving rather than TensorZero's gateway and LLMOps combination.

LLM Gateway offers a free-forever plan with access to more than 200 models across 40-plus providers, but includes a 5% charge on credit usage and free-model rate limits.

OpenRouter is a hosted API option with a free tier, 25-plus free models, 50 requests per day, and five workspaces; pick it when avoiding self-hosted operations outweighs database ownership.

Browse more options in LLM Gateway Software.

Verdict

TensorZero is best for an engineering-led team that wants a no-cost, self-hosted gateway with provider fallbacks, SQL-accessible inference data, and built-in experimentation. Those capabilities are unusually broad for a free package. The decisive reason to look elsewhere is its archived, unmaintained status: choose TensorZero only when your team is prepared to own the resulting maintenance risk.