How to Connect Any Self-Hosted AI Model to Pangolin

On-prem AI usually starts as a cost or data-sovereignty decision: run models on hardware you already own instead of paying per token, or keep certain data from leaving your network at all. It rarely stays that simple. Once a model server exists on your infrastructure, someone has to decide how the rest of the team reaches it, and that's a harder problem than picking an inference engine. Tools like vLLM, Ollama, llama.cpp, LM Studio, LocalAI all require you to figure out how to connected to them first.

The Problem With Reaching Self-Hosted Models

Operating a model gateway requires ongoing infrastructure management - whether deployed on Kubernetes, a VPS, or a spare machine - alongside routine patching like any other production service. It demands dedicated network reachability to the model server, which typically means placing them on the same network or maintaining a custom tunnel. On the user management side, engineers must generate and distribute individual API keys via scripts, email, or manual handoffs. Finally, every consuming client - from coding agents to chat applications - needs to be configured with a static key and a reachable gateway URL, usually exposed publicly or routed through a VPN. That's a lot of moving parts for what's conceptually a simple ask: let authorized people call a model that's running on a box you own.

The Pangolin Model: Identity Instead of Keys, a Tunnel Instead of a Network

Pangolin's AI gateway solves the same routing and access-control problem, but two things work differently.

Users install a desktop app and log in with your existing identity provider - Okta, Azure, Google, whatever you already run. The app does a device check and builds a WireGuard-based tunnel to Pangolin. No key touches the machine, because the client already proved who's calling. Nothing to generate, rotate, or worry about leaking.

The model server never has to be reachable from the internet or a shared network. A small connector runs next to the model, opens an outbound connection to Pangolin, and that's the only thing your infrastructure exposes. Whether the model is a single vLLM instance, an Ollama box, or a cluster behind an internal router, the same connector pattern applies.

Set Up Any Self-Hosted Provider

The steps are identical regardless of which inference engine you run - only the port and API shape change.

1. Confirm the API Format

Almost every self-hosted engine speaks OpenAI's Chat Completions API at some path:

A few engines speak other formats too - Kimi K2's own server is Anthropic-compatible, for instance. Pick whichever capabilities match what your server actually exposes; a Custom provider can select any combination.

2. Install a Site Connector Next to the Model

In the dashboard, go to Sites > Create, choose Newt as the site type, and name it. Pangolin generates a site ID, secret, and a ready-to-copy install command. Run that command on the model host, or any machine on the same network that can reach it:

curl -fsSL https://static.pangolin.net/get-newt.sh | bash
newt --id <site-id> --secret <site-secret> --endpoint https://app.pangolin.net

See Install Sites for the full reference.

This is the only software your infrastructure needs beyond the model server itself. It makes an outbound connection and holds it open - no inbound firewall rule anywhere.

3. Create a Custom Provider

AI Gateway > Providers > Create > Custom:

1. Name it for the server ("Internal vLLM", "LM Studio - Dev Box", etc.).
2. Select the API capability it speaks - usually OpenAI Chat Completions.
3. Routing Mode: Site Targets, so the request travels over the Newt tunnel rather than needing the Pangolin node to reach the host directly.
4. Auth Type: No Auth on a private network, or Bearer if the server has its own key.
5. Allow the exact model id(s) it serves.
6. Save, then add the target on the Network tab: host, port, HTTP or HTTPS.

Full field reference in Custom Providers. Worked examples for specific engines: Ollama, vLLM, Bifrost.

4. Attach to a Resource

Resources > Create > Type AI Gateway, attach the provider. Two reachability options:

• Private - only devices connected with the Pangolin client can call it. Identity comes from the tunnel; no API key exists to manage.
• Public - reachable from anywhere, authenticated with a virtual API key per user or a manual key for services.

5. Point Clients at the Resource

Every resource has copy-paste setup for common tools, and pangolin configure <client> writes the config automatically for supported ones:

pangolin configure claude --resource <resource>

For anything else that speaks OpenAI's API:

export OPENAI_BASE_URL=<endpoint>/v1
export OPENAI_API_KEY=none

See the full client setup guides for Claude Code, Codex, OpenCode, Gemini CLI, and Open WebUI.

Beyond the Basics

Once the provider is connected, the rest of what you'd expect from an AI gateway is already there:

• Mix local and cloud on one endpoint. Attach a self-hosted provider alongside OpenAI, Anthropic, or others on the same resource. Clients switch models without switching endpoints. See Model Routing for how the gateway picks between them.
• Split access by team. Different resources can expose different providers, models, and budgets to different roles - even from the same hostname.
• Budgets. Cap spend or token usage per provider, model, resource, role, or key. See Budgets.
• Session logs and usage analytics. Every call is attributed to a user or key, with prompt/response transcripts and cost breakdowns, without instrumenting the model server itself.
• Virtual keys for machines. Services, CI, and agents that can't run the desktop client get a manual key scoped to exactly the resources they need.

Self-Hosted or Cloud Control Plane

Everything above works whether you use Pangolin Cloud or run the control plane yourself. The Community Edition is open source under AGPLv3; the Enterprise Edition is also source-available under a commercial license that permits free personal and small-business use. Point Newt and the client at your own instance instead of app.pangolin.net and nothing else in this guide changes.

FAQ

How do I expose a self-hosted model to my team without opening a port?

Install a Pangolin site connector on the same network as the model server. It opens an outbound-only tunnel, so nothing on your infrastructure needs to accept inbound connections or have a public IP. Attach the model as a Custom provider and reach it through an AI Gateway resource.

Does this work with any inference engine?

Yes, as long as it exposes an HTTP API - vLLM, Ollama, llama.cpp, LM Studio, LocalAI, text-generation-webui, and downstream routers like Bifrost all work the same way. Select whichever capability matches the API format it speaks, most commonly OpenAI Chat Completions.

Can my team use this without API keys?

For private resources, yes. Users log in through the Pangolin desktop app with your existing identity provider, and that authenticates every call - no key is issued or required. Public resources, meant for reachability from anywhere, use virtual API keys instead of real provider credentials.

How is this different from running LiteLLM or Bifrost myself?

Those tools solve model routing well but leave key distribution and network reachability to you. Pangolin replaces static keys with identity-based access through a client app, and replaces "the gateway needs network access to the model" with an outbound tunnel from a small connector. You can still run Bifrost or LiteLLM behind Pangolin if you want their routing logic specifically.

Can I connect more than one self-hosted server to the same gateway?

Yes. Add each server as its own Custom provider and attach as many as you need to a resource, or split them across multiple resources for different teams. See Multiple Gateway Resources.

See Also

• AI Gateway Overview: the full provider, resource, and key model
• Custom Providers: field-by-field configuration reference
• NVIDIA DGX Spark Remote Access: this pattern applied to a single box
• vLLM and Ollama Clusters: this pattern applied to a multi-node cluster
• Self-Host DeepSeek, Qwen, and Kimi K2: open-weight models specifically
• How to Stop Using Virtual API Keys: moving human users to identity-based access instead of keys