vLLM and Ollama Clusters: Secure Remote Access Without a VPN

Once a team outgrows running models on a single box like a DGX Spark, they might deploy vLLM on a rack of GPUs for throughput and Ollama somewhere for smaller models and quick iteration, maybe both behind a router to pick between them. That solves model serving. It doesn't solve who gets to call the cluster, or how they reach it if they're not sitting on the same network as the GPUs.

Most teams end up doing one of two things. Either the cluster sits on a network everyone already has VPN access to, which means every engineer's laptop is one misconfigured client away from talking directly to GPU hosts. Or the cluster is only reachable from inside a data center or office, and remote work means SSHing into a jump box before anything useful happens. Both work but neither is something you'd design on purpose for good user experience and security.

What Managing This Yourself Looks Like

If you've deployed an AI on your own infrastructure you are familiar with some of the challenges. Operating the gateway requires hosting it on self-managed infrastructure-such as Kubernetes, a VPS, or a virtual machine-where you handle uptime, patching, and reachability. Wherever it runs, it needs network access to the vLLM and Ollama endpoints, whether by sitting on the same subnet or connecting via a secure tunnel. Providing engineer access involves generating API keys for each person or team and distributing them manually, through scripts, or via gateway-triggered emails. Finally, every engineer must configure all of their coding tools with the gateway’s reachable URL and store their assigned key in a local config file on their laptop.

None of that is wrong, exactly. It's just a lot of infrastructure and key management to stand between an engineer and a model that's already running on hardware you own.

Attach the Cluster to Pangolin Instead of Standing Up Your Own Gateway

Pangolin is an AI gateway that does the same job - one endpoint, multiple upstream model providers, access control, budgets, logging - but it authenticates callers by identity through a desktop client or by personal API keys instead of by handing out static keys to each person, and it reaches your cluster over an outbound tunnel instead of requiring your gateway and your GPUs to share a network.

1. Install a Site Connector Near the Cluster

In the dashboard, go to Sites > Create, choose Newt as the site type, and name it. Pangolin generates a site ID, secret, and a ready-to-copy install command. Run that command on a machine that can reach both the vLLM and Ollama endpoints - one of the cluster nodes, or a small VM on the same network:

curl -fsSL https://static.pangolin.net/get-newt.sh | bash
newt --id <site-id> --secret <site-secret> --endpoint https://app.pangolin.net

See Install Sites for the full reference.

This is the only thing installed on your network. Newt opens an outbound connection to Pangolin; nothing needs to accept inbound traffic from the internet.

2. Add vLLM as a Provider

vLLM's OpenAI-compatible server defaults to port 8000. In the dashboard, AI Gateway > Providers > Create:

1. Provider Type: Custom, name it for the cluster (e.g. "vLLM - Cluster A").
2. Select OpenAI Chat Completions.
3. Routing Mode: Site Targets.
4. Auth Type: No Auth on a private network, or Bearer with the key if you started vLLM with --api-key.
5. Allow the served model id - the --model or --served-model-name value.
6. Save, then add a target: the vLLM host, port 8000, HTTP.

Full reference: vLLM as a Custom provider.

3. Add Ollama as a Second Provider

Repeat with a second Custom provider for Ollama:

1. Select OpenAI Chat Completions.
2. Routing Mode: Site Targets.
3. Auth Type: No Auth, or Bearer if you set an Ollama API key.
4. Allow the model ids you serve, e.g. llama3.2, qwen2.5:32b.
5. Target: the Ollama host, port 11434, HTTP.

Reference: Ollama as a Custom provider.

4. Attach Both to One Resource

Resources > Create > Type AI Gateway, and attach both providers. One endpoint now routes to vLLM or Ollama depending on the model requested - the client just asks for a model id, and model routing picks the right provider. If a model id is served by both, allow-list scoping or selecting a subset per resource keeps it unambiguous.

If you're already running a downstream router like Bifrost in front of vLLM and Ollama, you don't have to give that up - point Pangolin at Bifrost as a single Custom provider instead, and let Bifrost keep doing model selection and failover behind Pangolin's identity layer.

5. Split Access by Team

Different roles rarely need the same models. Create separate resources instead of one big shared one. For example, you could break things up like this:

• Engineering resource: full model set on vLLM and Ollama, private, reached through the client.
• Data Science resource: same cluster, narrower allow list, maybe with a budget attached.
• CI / Agents resource: public, authenticated with a manual virtual key scoped to just that resource, for pipelines that can't run the desktop client.

See Multiple Gateway Resources - resources can even share a hostname, so the split is invisible to the person using it.

Connect Engineers

Everyone installs the Pangolin client and logs in with your existing IdP - Okta, Azure, Google, whatever you already run. No key to request, copy, or rotate. Coding agents and chat clients point at the resource URL:

pangolin configure claude --resource cluster-a.internal

or manually, for any OpenAI-compatible tool:

export OPENAI_BASE_URL=<endpoint>/v1
export OPENAI_API_KEY=none

Revoking someone's access is a permission change on the resource, not a key rotation across every host that had it.

Why This Beats Self-Managing the Gateway

• No gateway infrastructure to run. Pangolin Cloud handles the control plane, or self-host it if that's a requirement - Pangolin is open source under AGPLv3.
• No key distribution. Identity keys already exist for every org user; there's nothing to generate, email, or leak.
• The cluster stays private. vLLM and Ollama are reached over an outbound tunnel from Newt, never exposed to the network your engineers' laptops sit on.
• Vendor-agnostic on top. The same resource can also attach OpenAI, Anthropic, or any other provider, so switching between your cluster and a frontier model is a model-name change, not an endpoint change.
• Usage attribution for free. Session logs and usage analytics track who called what, without instrumenting vLLM or Ollama yourself.

FAQ

How do I give a remote team access to a self-hosted vLLM or Ollama cluster?

Install a site connector on the cluster's network, add vLLM and/or Ollama as Custom providers routed through it, and attach them to an AI Gateway resource. Engineers install the Pangolin client, log in with SSO, and reach the cluster over a tunnel - no VPN and no distributing API keys.

Can Pangolin route between vLLM and Ollama automatically?

Yes. Attach both as providers on the same resource. Model routing picks the provider whose allow list matches the requested model id, so clients just ask for a model - they don't need to know which backend serves it.

Do I still need LiteLLM or Bifrost if I use Pangolin?

No, but you can keep it. Pangolin replaces the identity and key-management layer, not necessarily your model router. If you already have Bifrost picking between providers, point Pangolin at Bifrost as a single upstream and let Pangolin handle authentication in front of it.

How do I keep the GPU cluster off the same network as engineer laptops?

Only the site connector needs network access to vLLM and Ollama. It makes an outbound-only connection to Pangolin, so engineers never join the cluster's network - they reach a Pangolin resource, which is scoped to the model APIs you've attached, not the underlying hosts.

Can I give a CI pipeline access without a person's identity?

Yes - create a manual virtual key scoped to a public resource. It authenticates independently of any user's grants, which is what unattended jobs need.

See Also

• AI Gateway Overview: providers, resources, and routing
• vLLM as a Custom Provider and Ollama as a Custom Provider
• Multiple Gateway Resources: split access by team without duplicating the cluster
• NVIDIA DGX Spark Remote Access: the same pattern for a single box
• Self-Host DeepSeek, Qwen, and Kimi K2: running open-weight models on this same kind of cluster
• Why Virtual API Keys Are a Bad Fit for LiteLLM and Bifrost Deployments: the credential problem this setup avoids