Inference Gateway
Inference Gateway is a free, open-source cloud-native proxy that presents hosted and local language-model providers through one OpenAI-compatible API.
What Inference Gateway is
Inference Gateway is a free, open-source cloud-native proxy that presents hosted and local language-model providers through one OpenAI-compatible API. Listed providers include OpenAI, Anthropic, Groq, Cohere, Ollama, DeepSeek, Google, Mistral, NVIDIA, and others. Applications can switch providers through configuration without redeploying, while model aliases can route to pools of upstream deployments. MCP support discovers tools from connected servers and executes calls through the gateway. Agent-to-Agent features support capability discovery, delegation, and streamed results; agents can be defined in YAML, with a CLI generating Go, Rust, or TypeScript projects. A Kubernetes Operator manages gateways, agents, MCP servers, and chat orchestrators. Documentation covers OIDC authentication, Prometheus metrics, OpenTelemetry, structured JSON logs, and Grafana, Loki, and Jaeger examples. Docker and Kubernetes deployment are documented, and self-hosting can run on-premises, in the cloud, or air-gapped. The project uses the Apache-2.0 license.
Good
- One OpenAI-compatible API
- Switch providers without redeploying
- Supports MCP server connections
- Kubernetes Operator included
- Self-hosted and air-gapped deployment
Watch out
- Requires deployment and infrastructure management
- Desktop releases lack Apple and Windows signing
- No fallback routing listed
- Provider setup varies by configuration
Who it suits
Inference Gateway suits developers and platform teams operating multiple hosted or local LLM providers. It fits deployments needing self-hosting, Kubernetes management, MCP tools, agent coordination, and observability.
Verdict
Inference Gateway centralizes provider access while adding routing, MCP, agent, Kubernetes, and observability features. Its open-source Apache-2.0 model and self-hosting options are suited to teams managing their own infrastructure.
The facts
Each one read from Inference Gateway's own pages, with the date it was checked.
What it does
- Agent coordination
- Its Agent-to-Agent support lets agents discover capabilities, delegate tasks, and stream results within a conversation.checked 10 Oct 2026
- Agent workflows
- The product supports Agent-to-Agent coordination and describes defining agents in YAML with a CLI that generates Go, Rust, or TypeScript projects.checked 9 Oct 2026
- Authentication
- Authentication supports OIDC identity providers, with JWT validation against the issuer’s JWKS.checked 10 Oct 2026
- Deployment
- The getting-started guide documents Docker, Docker Compose examples, and Kubernetes deployment.checked 10 Oct 2026
- Footprint
- The gateway is described as an approximately 10.8 MB static binary designed to scale horizontally with Kubernetes HPA.checked 10 Oct 2026
- Kubernetes
- A Kubernetes Operator manages gateways, agents, MCP servers, and chat orchestrators declaratively as cluster resources.checked 10 Oct 2026
- License
- The documentation states that Inference Gateway is released under the Apache-2.0 license.checked 9 Oct 2026
- License and support
- The project is released under Apache-2.0, with GitHub Discussions for questions and an issue tracker for bugs and feature requests.checked 10 Oct 2026
- MCP integration
- It can discover tools from connected MCP servers and execute tool calls server-side.checked 10 Oct 2026
- Observability
- Built-in observability includes Prometheus metrics, OTLP tracing, structured JSON logs, and reference Grafana dashboards.checked 10 Oct 2026
- Privacy
- The project states that it has no analytics or telemetry phoning home and can be self-hosted on-premises, in the cloud, or air-gapped.checked 10 Oct 2026
- Product
- Inference Gateway is an open-source, cloud-native proxy that presents supported hosted and local LLM providers through one OpenAI-compatible API.checked 9 Oct 2026
- Provider coverage
- The documentation lists providers including OpenAI, Anthropic, Groq, Cohere, Ollama, DeepSeek, Cloudflare, Google, Mistral, MiniMax, Moonshot, Nvidia, and llama.cpp.checked 10 Oct 2026
- Provider switching
- The gateway lets applications switch providers by changing configuration, without an application redeploy, and can route model aliases to pools of upstream deployments.checked 9 Oct 2026
- Purpose
- Inference Gateway is an open-source, cloud-native proxy that provides one OpenAI-compatible API for multiple hosted and local LLM providers.checked 10 Oct 2026
- Routing
- The gateway supports model routing so a stable model alias can point to a pool of upstream deployments.checked 10 Oct 2026
- SDKs
- Official SDKs are available for Python, TypeScript, Go, and Rust, with streaming support over Server-Sent Events.checked 10 Oct 2026
Where it runs
- Desktop platforms
- The desktop app page lists Linux, macOS, and Windows assets for amd64 and arm64; it also says releases are not signed with Apple Developer or Windows code-signing certificates.checked 9 Oct 2026
Alternatives to Inference Gateway
- 1
Vellum
Free planOpen sourceAndroidiOSWebAPI8.7From $30 a month - 2
Dify
Free planOpen sourceWebAPIExtensionLinux8.4From $590 a year - 3
Inngest
Free planOpen sourceWebAPILinuxmacOS8.3From $99 a month - 4
Liner
Free planAndroidiOSWebAPI8.1From $14.99 a month - 5
Rasa
Free planOpen sourceWebAPILinuxmacOS7.8Free plan - 6
agentgateway
Free planOpen sourceWebAPILinuxmacOS7.5Free plan