Why Virtual API Keys Are a Bad Fit for LiteLLM and Bifrost Deployments

LiteLLM and Bifrost rout requests across many model providers behind one API, with budgets, rate limits, and fallback logic that a raw provider SDK doesn't give you. Neither one is the issue here. The issue is the credential handling both of them inherited from the APIs they proxy - the virtual API key - and what happens when a team quietly starts using that key as a stand-in for identity.

Virtual keys do exactly what they're designed to do: authenticate a caller to an HTTP API. The trouble starts when an engineering team treats "has a valid key" as equivalent to "is an authorized person." Those are very different security considerations and conflating them is where most of the pain in a self-hosted AI gateway deployment comes from.

What People Are Doing Now

You might stand up LiteLLM or Bifrost on your own infrastructure - Kubernetes, a VPS, wherever - because you want one endpoint in front of OpenAI, Anthropic, and a self-hosted model cluster, plus budgets and usage tracking. That part works fine. Then it needs to reach whatever's actually serving the local models, which usually means it lives on the same network as the cluster, or someone builds a tunnel so it can. Now the gateway is also a network problem, not just a routing one.

Then people need to use it. Someone writes a script - or clicks through an admin panel - to mint a virtual key per engineer. Those keys get emailed, dropped in a wiki page, or sent over Slack, because there isn't a cleaner distribution channel built in. Each engineer pastes their key into every coding tool and chat client they use, and the gateway's URL has to be reachable from wherever they're working - which means it's either public on the internet, or everyone's also running a VPN to get to it. None of these are unreasonable decisions in isolation. Together, they add up to a system where "who can use our AI gateway" is answered by "whoever still has a copy of a key," which is a much weaker answer than most teams realize they're settling for.

Keys vs Identities

A few consequences follow directly from keys standing in for identity:

Keys outlive the reason they were issued. A key minted for a contractor, a short-term project, or someone who's since left the team doesn't expire on its own. Someone has to remember it exists and revoke it. In practice, that tracking degrades the same way any manual process does - slowly, and mostly unnoticed until an audit or incident forces the question.

Keys travel further than intended. A key pasted into a coding agent's config file, an environment variable, a CI job, or a support ticket doesn't stay where it was put. It gets copied into a dotfiles repo, logged by a tool that wasn't supposed to log secrets, or shared in a thread with someone who didn't need it. None of that requires anyone to act maliciously. Keys just spread the way any long-lived plaintext credential does.

Revocation is a rotation problem, not a permission change. Pulling access from one person by rotating a shared or role-scoped key means rotating it for everyone who legitimately still needs it, then redistributing the new one through whatever informal channel got used the first time. People respond to that friction by not rotating, which reintroduces the first problem.

The gateway has to be reachable, which conflicts with the reason a lot of teams self-host in the first place. If the whole point of running your own model cluster is to keep data off the internet, putting the gateway on a public URL - or asking everyone to VPN in - works against that goal. A VPN, in particular, tends to hand out much more network access than "reach the AI gateway" actually requires.

Usage attribution is only as good as key hygiene. LiteLLM and Bifrost both track spend and usage per key, which is genuinely useful - budgets and cost dashboards work. But a shared key, or a key that's been handed off between people, attributes activity to whoever holds the string, not to whoever's actually making the call.

It's worth being precise about what's being criticized here. LiteLLM and Bifrost aren't missing a "generate keys more securely" feature. The problem is an API key is, by design, a bearer credential. Whoever holds it can use it. There's no layer underneath it that ties the credential to a verified person, a device, or an organizational identity that has its own lifecycle - onboarding, role changes, offboarding - independent of whether someone remembers to touch the key. A public, internet-reachable endpoint calling itself an AI gateway more or less has to accept a bearer credential from unauthenticated callers - there's no other way to prove who's calling over a plain HTTP request.

What Identity-Based Access Changes

The alternative is to stop making the key the thing that proves who's calling, and let a verified identity do that instead. A user installs a desktop client and logs in with the identity provider your org already runs - Okta, Azure, Google, whatever handles onboarding and offboarding today. The client does a device check and builds a private, WireGuard-backed tunnel to the gateway. When a request comes through that tunnel, the gateway already knows exactly who's calling, because the network path itself is the proof. There's no key to generate, distribute, rotate, or leak, because there's no key in the loop for that request at all.

This doesn't mean throwing away everything a virtual key is good for. Services, CI pipelines, and agents that can't run a desktop client still need something to authenticate with, and a scoped, budgeted key is the right tool for that. It's the human side of the equation where treating a key as an identity stops making sense. Pangolin still supports both: private resources authenticated by client identity for people, and virtual keys scoped for machines that genuinely need them. The difference is that identity is the default, and a key is the exception, not the thing every user ends up depending on because it was the fastest way to wire things up.

You can keep LiteLLM or Bifrost in this picture, too. Point Pangolin at either one as a Custom provider and let it keep doing model routing and failover - Pangolin only has to sit in front, handling who gets to call in the first place.

None of this requires ripping out your gateway or your provider setup - just changing what sits in front of it and decides who's allowed to call. If you want the walkthrough, How to Stop Using Virtual API Keys covers the migration step by step. Or get started with Pangolin and see what it looks like with your own LiteLLM or Bifrost deployment behind it.