Envoy AI Gateway

Agent Router is an open-source Envoy-based control plane for AI models, agents, and MCP tools. It fits teams that share providers or tools and need one place for credentials, routing, quotas, and attribution. The strongest reason to choose it is broad, self-managed governance; the main trade-off is that serious production operation is tied to a Kubernetes and Envoy Gateway stack.

Overview

The project exposes an OpenAI-compatible API for both hosted and self-hosted models, plus MCP servers. Its integrations cover OpenAI, Azure OpenAI, Google Gemini, Vertex AI, AWS Bedrock, Mistral, Cohere, Groq, Together AI, DeepInfra, DeepSeek, Hunyuan, SambaNova, Grok, and Anthropic. That makes it a practical common front door for mixed-provider estates, rather than an inference runtime or model marketplace.

Founded in 2024, it is now an Agentic AI Foundation project under the Agent Router name. The rename leaves the code, maintainers, and APIs unchanged. Apache 2.0 licensing permits self-managed use, while support is handled through community Discord and GitHub issues.

Key features

Traffic and budget controls

Model virtualization lets applications address models through a consistent interface, while provider failover can keep traffic moving when an upstream is unavailable. Token quotas and usage-based rate limits help prevent one workload from consuming a shared allocation. Usage analytics, usage attribution, and budget controls give administrators a way to understand and constrain consumption. Those controls are valuable for organizations with multiple teams; a small application needing a thin forwarding proxy may find the wider policy surface unnecessary.

MCP gateway

The MCP gateway multiplexes servers, routes tools, supports OAuth authentication, and provides observability. It is designed for teams exposing several tool back ends through one managed entry point, with routing and identity handled centrally instead of separately in every agent.

Inference routing

InferencePool support adds intelligent routing and load balancing for inference endpoints. This is a better fit for fleets of endpoints than for a single local model, where the added control layer may not justify itself.

Credential security and monitoring

Upstream authentication protects connections to AI services while keeping provider secrets out of application workloads. Central management includes short-lived credential flows for AWS Bedrock, Azure OpenAI, and GCP Vertex AI. Metrics collection and workload monitoring support operational oversight, particularly when several providers and teams share the gateway.

Pricing

There is no paid tier or hosted seat allowance in this plan. The cost is zero, but infrastructure, deployment, and operations remain your responsibility. Local use is available through the aigw run CLI; production Kubernetes operation requires Agent Router v1.2.x with Kubernetes v1.32 or later, Envoy Gateway v1.9.2 or later, and Gateway API v1.6.1.

Platforms

Agent Router supports API access, Linux, macOS, and self-hosted deployment. The local CLI currently runs on Linux and macOS. It can operate as a standalone local proxy or be deployed to Kubernetes with Envoy Gateway; the project also points to hosted services supplied by other providers.

Who it's for

This is aimed at teams sharing models or MCP tools that need centralized credentials, quotas, usage attribution, failover, analytics, or spending controls. Platform and infrastructure groups running several inference endpoints will benefit most from InferencePool routing and shared policy. It is less suitable for users who want a managed, vendor-operated gateway or a simple single-provider API with no Kubernetes operations.

Pros and cons

Pros

• Broad provider coverage: one OpenAI-compatible surface spans major commercial and self-hosted options.
• Strong governance: quotas, rate limits, budgets, analytics, and attribution address shared-use accountability.
• MCP-aware gateway: multiplexing, tool routing, OAuth, and observability target agent workloads directly.
• Credential isolation: centralized and short-lived provider credentials reduce the need to place secrets in applications.
• Open-source economics: Apache 2.0 software is free to run and can be self-hosted.

Cons

• Production dependency chain: Kubernetes, Envoy Gateway, and Gateway API versions must align for the documented deployment.
• Operational burden: self-hosting shifts infrastructure, upgrades, and reliability work to your team.
• Narrow local OS range: the CLI currently supports Linux and macOS, not a broader desktop set.
• Community-based support: assistance comes through Discord and GitHub issues rather than a stated commercial support tier.

Alternatives

Bifrost is the better comparison for teams wanting a freemium gateway with web and Windows support alongside self-hosting, including a free Apache 2.0 OSS plan.

Routerly suits users seeking a free self-hosted option with no markup on API calls and an AGPL-3.0 license.

YoloRouter is an alternative free self-hosted single binary with SQLite included and optional PostgreSQL.

Kong Gateway is worth considering when a paid gateway with a 30-day, no-card trial and web support is preferable.

TensorZero offers a free, fully self-hosted open-source LLMOps platform.

SGLang is the alternative for an open-source inference framework installed with pip or Docker.

LLM Gateway fits readers wanting a freemium service with 200-plus models across 40-plus providers and a hosted free tier.

OpenRouter is the simpler API-and-web choice for credit-based access to free models and providers without self-hosting.

Browse more options in LLM Gateway Software.

Verdict

Choose Agent Router if your team needs an open-source, Envoy-centered control plane for shared models and MCP tools, especially with centralized credentials, failover, quotas, and analytics. Look elsewhere if you want managed hosting, broad local desktop support, or to avoid Kubernetes and its dependency requirements.