OpenZiti LLM Gateway
OpenZiti LLM Gateway is a free, Apache 2.0 open-source API proxy for routing requests to OpenAI, Anthropic, and other OpenAI-compatible backends.
What OpenZiti LLM Gateway is
OpenZiti LLM Gateway is a free, Apache 2.0 open-source API proxy for routing requests to OpenAI, Anthropic, and other OpenAI-compatible backends. Clients use an OpenAI-compatible API, while the gateway can translate requests for Anthropic’s Messages API. Optional semantic routing combines keyword heuristics, embedding similarity, and an LLM classifier when no model is specified. Chat completions support Server-Sent Events streaming. Weighted round-robin balancing, health checks, fallback routing, and automatic failover distribute requests across endpoints. Virtual API keys can restrict clients to model patterns; health and metrics endpoints remain unauthenticated. Optional Prometheus metrics cover requests, latency, tokens, routing, provider errors, and endpoint health. The project describes one Go binary with YAML configuration, no database, queue, or sidecar, and self-hosted use on Linux, macOS, or Windows.
Good
- Free Apache 2.0 open source
- Supports OpenAI-compatible clients
- Offers semantic and fallback routing
- Includes load balancing and failover
- Provides Prometheus metrics
Watch out
- Health and metrics endpoints remain unauthenticated
- Tracing may log truncated message content
- Requires self-hosted deployment
Who it suits
It suits teams needing a self-hosted gateway for model routing, provider translation, fallback, usage analytics, budget controls, or virtual keys. The project specifically addresses gateways, GPU servers behind NAT, and clients needing access without opening ports.
Verdict
OpenZiti LLM Gateway offers a broad routing and networking layer in a single configurable binary. Its self-hosted model and unauthenticated health and metrics endpoints should be considered during deployment planning.
The facts
Each one read from OpenZiti LLM Gateway's own pages, with the date it was checked.
What it does
- Anthropic support
- The gateway translates between OpenAI format and Anthropic’s Messages API.checked 9 Oct 2026
- Client authentication
- Optional virtual API keys can be restricted to models using glob patterns; the health and metrics endpoints remain unauthenticated.checked 10 Oct 2026
- Deployment
- The project describes deployment as one Go binary and one YAML file, with no database, message queue, or sidecar required.checked 10 Oct 2026
- Integrations
- Documented backends include OpenAI, Anthropic, Ollama, vLLM, llama-server, and SGLang; clients use an OpenAI-compatible API.checked 10 Oct 2026
- Intended users
- The README describes routing across local inference servers and remote providers, including GPU servers behind NAT and clients using OpenAI-compatible libraries.checked 10 Oct 2026
- License
- The repository identifies the project as Apache 2.0 licensed.checked 10 Oct 2026
- Load balancing
- It supports weighted round-robin across inference endpoints, health checks, and failover.checked 10 Oct 2026
- Metrics
- When enabled, its Prometheus endpoint reports request, latency, token, routing, provider-error, and endpoint-health metrics.checked 10 Oct 2026
- Network security
- The project describes using zrok over OpenZiti to connect gateway and backend across NAT or network boundaries without firewall rules or port forwarding.checked 9 Oct 2026
- Observability
- When enabled, Prometheus-exported metrics include request counts, latency, token counts, routing decisions, provider errors, and endpoint health.checked 9 Oct 2026
- Provider integrations
- The project documents routing to OpenAI, Anthropic, Ollama, vLLM, llama-server, SGLang, and other OpenAI-compatible backends.checked 9 Oct 2026
- Provider translation
- It translates between the OpenAI request format and Anthropic’s Messages API.checked 10 Oct 2026
- Purpose
- It is an OpenAI-compatible API proxy that routes requests to OpenAI, Anthropic, and compatible backends such as Ollama and vLLM.checked 10 Oct 2026
- Routing
- Optional semantic routing uses keyword heuristics, embedding similarity, and an LLM classifier to choose a model when the request omits one.checked 9 Oct 2026
- Semantic routing
- An optional cascade of keyword heuristics, embedding similarity, and an LLM classifier can select a model when the request omits one.checked 10 Oct 2026
- Streaming
- It supports server-sent event streaming for its providers.checked 10 Oct 2026
- Tracing
- Optional tracing logs request metadata and truncated message content, with a configurable maximum content length.checked 9 Oct 2026
- Zero-trust networking
- With zrok over OpenZiti, the gateway can expose itself or reach backends without firewall rules or port forwarding.checked 10 Oct 2026
The company
- Headquarters
- Charlotte, North Carolina, United Stateschecked 28 Sep 2026
Where it runs
- Platforms
- The README says prebuilt binaries are available for Linux, macOS, and Windows, and also documents installation with Go or from source.checked 10 Oct 2026
Alternatives to OpenZiti LLM Gateway
- 1
LangSmith
Free planWebAPILinuxmacOS8.0$39 usage-based - 2
Manifest
Free planOpen sourceWebAPILinuxSelf-hosted7.9From $19 a month - 3
Bifrost
Free planOpen sourceWebAPILinuxmacOS7.6Free plan - 4
TrueFoundry AI Gateway
Free planWebAPILinuxSelf-hosted7.6From $499 a month - 5
YoloRouter
Free planOpen sourceWebAPILinuxmacOS7.5Free plan - 6
Eden AI
WebAPI7.4$5.50 one-time