Best Open Source AI Gateways in 2026

This is a comparison of the best open source and self-hosted AI gateways in 2026: LiteLLM, Bifrost, Pangolin, Portkey Gateway, and Kong AI Gateway. An AI gateway sits between coding agents, apps, and pipelines on one side and model providers on the other. It authenticates the caller, picks an upstream, tracks spend, and records what ran.

What Is a Self-Hosted AI Gateway?

A self-hosted AI gateway is a proxy you run on your own infrastructure. Clients call your URL instead of OpenAI, Anthropic, Gemini, or a local engine. The gateway forwards the request with the organization's real upstream credential, applies policy, and logs the call.

Open source means you can inspect and modify the software. Self-hosted means you can run it on your own infrastructure, so data stays on networks you already trust. When you self-host, you also operate the full process: upgrades, pooling, high availability, database, etc.

Open Source AI Gateways at a Glance

LiteLLM

LiteLLM is a dedicated LLM proxy. It presents a unified OpenAI-compatible API in front of many model providers, with virtual keys, spend tracking, fallbacks, and load balancing. Teams run it as a self-hosted proxy or call its SDK from application code. Clients point at one gateway URL for the instance.

The MIT-licensed Python SDK you can integrate into your application code is the usual reason to use LiteLLM, and it is the fastest way to get a working gateway with a huge provider list.

You still have to supply everything around the proxy. Identity is a virtual key you mint, distribute, and revoke, which can be challenging for large numbers of users. Reaching a GPU box on a private network is your networking problem. SSO, OIDC, and SCIM sit outside the MIT core.

LiteLLM fits a dedicated Python proxy or SDK in front of many providers, when you already have a network path and your own identity layer.

Bifrost

Bifrost is a dedicated LLM gateway focused on complex routing rules, failover, and high performance. It is Apache 2.0, written in Go, and self-hosts as a single binary with one gateway URL. Teams pick it for throughput, semantic caching, and MCP support. An example of a complex routing rule is a gateway that only allows users to call models in a specific region during a specific time of day. It advertises sub-millisecond routing overhead at high request rates.

Authentication is still a virtual API key. Like LiteLLM, Bifrost assumes you can already reach the upstreams it routes to. You can also run it downstream of Pangolin as a Custom provider when Pangolin should authenticate callers and Bifrost should handle complex routing rules.

Bifrost fits a dedicated high-performance LLM gateway for complex routing rules and failover, either on its own or running behind Pangolin.

Pangolin

Pangolin is an open source AI gateway: put one URL in front of many providers, with model routing, usage tracking, and access control. You can create many of those URLs on one platform, each with its own hostname, providers, models, and access rules. Identity, tunnels, and policy use the same users, roles, and connectors you already use for apps and other infrastructure you protect with Pangolin.

You grant users and roles the same way you do for other Pangolin resources. On a public AI Gateway, coding agents send a virtual API key. That key identifies the user. You grant the user or role. On a private AI Gateway, the Pangolin client already authenticated that user. Coding agents on that device call over the tunnel, and Pangolin attributes the request to the connected identity. Manual keys remain for services, CI, and machines that cannot run the client.

Pangolin supports all public cloud providers, like OpenAI, Anthropic, Gemini, Bedrock, Vertex, Foundry, OpenRouter, Vercel AI Gateway, and whatever HTTP endpoint you attach. Tunneling works on both sides which also makes it easy to reach self-hosted models. A site connector sits on the network where Ollama, vLLM, or another model server already runs. Custom providers use site targets so the gateway reaches that API over the tunnel. The model host does not need a public IP. Users connect with the client, then reach a private AI Gateway the same way they reach a private host or SSH resource. Each AI Gateway is its own resource with its own hostname, providers, models, and access rules.

The fuller three-way comparison is available at Pangolin vs. Bifrost vs. LiteLLM. Pangolin fits when you want the gateway itself to handle identities, tunnels to self-hosted models, and a client on end user devices to keep upstream keys off their devices. Pangolin makes it easy to eliminate virtual API keys from your stack while supporting all the usual AI gateway features.

Portkey Gateway

Portkey Gateway is a dedicated LLM gateway focused on guardrails. PII detection, jailbreak checks, and other input and output policies run on the request path. The gateway can then deny the call, retry, or fall back to another model. An example is stripping secrets from a prompt before it reaches OpenAI, or refusing a completion that looks like a jailbreak. It is MIT-licensed TypeScript, with a large provider catalog, retries, fallbacks, load balancing, and caching. Run the open-source binary with npx @portkey-ai/gateway and point clients at that one URL.

That binary is not the full product. Dashboards, request-level observability, cost tracking, prompt versioning, longer log retention, and most of the guardrail catalog sit on a hosted control plane. You can keep the proxy on your own infrastructure but it still connects back to that hosted plane, therefore Portkey is not a fully self-hosted gateway.

Portkey fits when guardrails on the routing path are the main job, and a hosted control plane is acceptable.

Kong AI Gateway

Kong AI Gateway extends Kong Gateway with LLM routing, MCP, and agent traffic. Kong Gateway itself is Apache 2.0 and widely self-hosted. LLM traffic is another set of routes that can be added to the same Kong instance. If your organization already runs Kong for APIs, you might consider using Kong AI Gateway instead of starting from scratch.

Kong's current AI Gateway 2.x is documented and sold primarily through Konnect, with an on-prem option. Many of the AI-specific plugins teams actually need (advanced proxy behavior, PII redaction, semantic caching, enterprise SSO) have historically sat in Kong Enterprise rather than in the Apache 2.0 core.

Kong AI Gateway fits teams that already operate Kong. Starting from scratch with Kong just to get an LLM proxy is usually more platform than you need.

Can You Run LiteLLM or Bifrost Behind Another Gateway?

Yes. Put LiteLLM or Bifrost on the network that already runs your models, then attach it as a Custom provider in Pangolin. Callers hit a Pangolin URL you choose. Pangolin authenticates them and tunnels to the private network. The downstream gateway keeps doing model routing and failover.

That split is useful when you like LiteLLM's catalog or Bifrost's routing rules and want identity in front of the router. The walkthrough is How to Stop Using Virtual API Keys. The reasoning is in Why Virtual API Keys Are a Bad Fit for LiteLLM and Bifrost Deployments. The three-way comparison is Pangolin vs. Bifrost vs. LiteLLM.

Final Thoughts

An open source AI gateway is worth running when you want a URL of your choosing in front of several providers, a place to cap spend, and a record of who called what. LiteLLM, Bifrost, Portkey, and Kong each give you one of those URLs per instance. Pangolin lets you create many, with different rules, on one platform. Self-hosting is worth it when prompts, keys, or model weights should stay on infrastructure you operate or you want to control costs.

FAQ

What is the best open source AI gateway?

It depends which constraint bites first. LiteLLM has the widest provider catalog. Bifrost is built for routing performance. Portkey is built for guardrails, with a hosted control plane around an MIT proxy. Kong fits teams that already run Kong. Pangolin fits when you want to manage identities, tunnels to self-hosted models, and serve many gateway URLs with their own rules.

Can you self-host an AI gateway?

Yes. LiteLLM, Bifrost, Pangolin, Portkey Gateway, and Kong Gateway all run on your own infrastructure. Pangolin also offers a hosted cloud offering if you want to avoid self-hosting.

Is LiteLLM fully open source?

The proxy core is MIT-licensed and self-hostable, including routing, virtual keys, and spend tracking. SSO, OIDC, and SCIM sit outside that core. That split shows up across this category.

What is the difference between LiteLLM and Bifrost?

LiteLLM is a Python-based LLM proxy with the largest provider list and the fastest path to a working self-hosted endpoint. Bifrost is a Go-based gateway aimed at throughput, routing rules, and failover, with a smaller catalog. Both expose one URL per instance, authenticate people with virtual API keys, and require manual networking to any self-hosted models they call.

How is Pangolin different from LiteLLM or Bifrost?

All three are AI gateways. LiteLLM and Bifrost each expose one URL for the instance, authenticate callers with virtual keys, and expect you to supply the network path to private models. Pangolin does the same routing and spend tracking, and lets you create many gateway URLs on one platform, each with its own rules, plus SSO identity, a client users run to authenticate themselves so they don't need API keys, and site connectors to reach self-hosted models on private networks. The full comparison is Pangolin vs. Bifrost vs. LiteLLM.

Can you keep LiteLLM or Bifrost and still use Pangolin?

Yes. Attach either one as a Custom provider. Pangolin authenticates and tunnels. The downstream gateway keeps routing, budgeting, and failover. See How to Stop Using Virtual API Keys.

Is Kong AI Gateway open source?

Kong Gateway is Apache 2.0. Kong's AI Gateway product is Konnect-first, with an on-prem option. Many AI-specific plugins that matter for production LLM routing have historically required Kong Enterprise. Check which plugins you need before treating it as a fully open-source AI gateway.

When is a managed AI gateway a better fit than self-hosting?

When you would rather not operate the proxy, its database, caching, and on-call rotation. Self-hosting wins when you need the request path, prompts, or model weights to stay on infrastructure you control, or when a policy keeps every component in-house.

See Also

• What Is an AI Gateway?: what the category is, and how it differs from API, LLM, and MCP gateways
• Pangolin vs. Bifrost vs. LiteLLM: the deeper three-way
• Why Virtual API Keys Are a Bad Fit for LiteLLM and Bifrost Deployments: keys vs identity
• How to Stop Using Virtual API Keys: moving people onto client identity
• How to Connect Any Self-Hosted AI Model to Pangolin: site tunnels to vLLM, Ollama, and custom endpoints
• AI Gateway Overview: providers, resources, keys, budgets, and logs