How We Built A Highly Available Reverse Proxy

The Problem With Being One Box

A single Pangolin instance contains a lot of state. It holds open WireGuard tunnels to every connected site. It keeps a websocket open to every site and client, waiting to push updates. Its Gerbil process knows, at any given moment, exactly which domains it can route to locally. None of that is written down anywhere else - it lives in the process, and if the process dies, so does the map.

That's fine for one box. It's a real problem the moment you want two. Run a second Pangolin instance next to the first and you don't get a bigger Pangolin but instead strangers who happen to share a name. A site connected to node A is invisible to node B. A client with a websocket open to node A can't be reached by a message that originates on node B. To a user, this all needs to look like one system sitting behind one domain but underneath it's two different servers that don't share memory and can independently crash, restart, and rejoin at any moment.

Getting there meant sorting state into three buckets: what gets promoted to something shared and durable (PostgreSQL), what stays fast and ephemeral but still cross-node (Valkey/Redis), and what has to stay local no matter what - and then building a routing layer smart enough to find the right node whenever it hits that last bucket.

The Easy Part Is The UI and API

The dashboard, the API, and DNS resolution turned out to be the straightforward part. Any node can authenticate a user or answer a resource query, because the things it needs - accounts, resources, access policies, certificates - already live in the shared database. Put a load balancer in front of every node, health-check it, and a dead node simply stops receiving traffic.

Certificate Generation

Every node can independently notice a domain needs one, but you don't want every node independently talking to Let's Encrypt. Two nodes racing the same DNS-01 challenge is a good way to get rate-limited or to mess up each other's in-flight validation. So exactly one node in a cluster is designated to issue and renew certificates. It stores them encrypted in the shared database once they're created. Every other node just reads them back out and drops them onto its own local disk, because Traefik on each node can only load certificates from files. It's an oddly analog step in an otherwise fully networked system, but it's the one that makes the certificate story boring: one writer, everyone else a reader.

Having the certificates in the database also means we can pull them out for other means - like sending them down to site connectors to serve private HTTPS resources over the VPN. Or to be sent out to remote nodes for things like site to cloud backhaul networking. ALl of these remote services need the certs and storing them centrally allows us to distribute them.

Clustering The Websocket

Every Pangolin VPN client and site connector keeps a persistent websocket open to whichever node it decides to connect to. That connection lives entirely in that node's memory because a single server needs to terminate the other end of that websocket. So when something on node A needs to push a message to a client that's actually sitting on node B's socket, node A has no direct way to reach it.

The fix is a shared pub/sub channel that every node listens to like Redis or Valkey. A node trying to reach a client checks locally first; if the client isn't there, it publishes the message instead, and whichever node actually holds that connection picks it up and delivers it. It lets the cluster treat "a message to this client" as a single operation regardless of which of the nodes is actually holding the socket at that instant.

Sites Pick Their Own Node

When a site connects - or reconnects after a failure - it pings a /ping endpoint on every node in the cluster and measures how fast each one answers. It connects to whichever node comes back fastest.

That's a deliberately simple mechanism, and it produces a useful behavior which is regional routing without anyone configuring regions. You could run one node in Virginia and one in Frankfurt, and a site sitting in Germany doesn't need to be told to prefer the European node because it measures a few milliseconds to Frankfurt against well over a hundred to Virginia and picks the obvious winner on its own. During a failover a node cant respond to the ping anymore and the site reconnects to whoever's alive now. When that happens, the node the site lands on tells the rest of the cluster it now holds that tunnel, and the DNS server starts answering resource queries for that site with the new node's address on the very next lookup.

DNS Handles Routing

Most systems treat DNS as a boring, mostly-static lookup - a hostname points at a server, and that mapping changes rarely enough that a five-minute TTL is good. That assumption doesn't hold here. The right answer to "what IP does this domain resolve to" depends entirely on which node currently holds the tunnel to the site serving that resource, and that can change the moment a site reconnects.

We couldn't bolt this onto someone else's DNS provider and call it done. A third-party DNS service can update a record when you tell it to, but it has no idea when a site moves from one node to another. So Pangolin runs its own authoritative DNS server, embedded in the same process that already knows in real time which node is holding which tunnel. A query for a resource's domain gets answered from the live state of the cluster. It's also why DNS delegation is a hard requirement for clustering rather than a nicety: you point an NS record at the cluster and hand it authority over a subdomain, because the entire point is that Pangolin has to be the one answering, not a static provider sitting a step removed from what's actually happening.

Teaching Gerbil to Ask for Directions

But what happens if the DNS record is stale? Every node runs its own Gerbil, and each one only knows the domains for sites tunneled directly to it. DNS returns whichever node the site is connected to, and browsers cache DNS answers for some time. So before Gerbil can do anything else with an incoming connection, it has to answer one question: do I actually have this site, or does one of the other nodes?

The trick is the same shape as one we used solving NAT traversal: make a routing decision off a header without fully processing what's behind it. Gerbil peeks at the hostname in the TLS handshake - the one plaintext field a connection exposes before encryption actually starts - before the handshake completes. If that domain belongs to a site tunneled to this node, it hands the raw connection to its local reverse proxy Traefik and gets out of the way. If it doesn't, it asks the control plane which node does have it, opens a plain connection to that node, and pipes the original, still-encrypted bytes straight through. The relaying node never touches the certificate or the traffic inside - it's just a dumb forwarder for an encrypted stream it can't read, which means the actual TLS session is still exactly between the client and whichever node holds the tunnel. Since that relay hop would otherwise hide the real client's IP address behind the relaying node's, each node tells its peers to trust a small header that carries the original address along for the ride.

This relay is hopefully the exception. The DNS behavior described above already resolves each resource to whichever node currently holds the tunnel, so most requests reach the right node directly. The relay exists for the small number that don't either because a client with a stale cached DNS answer, or a request that arrives mid-failover, before the new answer has had a chance to propagate.

Failing Over Without Flapping

When a node dies the load balancer notices almost immediately and just stops sending it anything, which is why the dashboard and API should not register much of a change. The sites connected to that node notice slower, because they were never talking through a load balancer in the first place - they hold a direct tunnel to one node's Gerbil, and that tunnel doesn't magically route around a dead process. It has to time out, notice, and go looking for a new home.

When it does, it doesn't just retry the node it lost - it's handed a list of healthy alternatives, checks which one it can actually reach well, and reconnects there. We deliberately bias that choice away from the node it just fell off of, on the theory that a node that failed ten seconds ago probably hasn't un-failed yet. We also deliberately damp the choice so a client doesn't jump nodes every time two options are neck-and-neck on latency - failover should be a decision made once when it's actually warranted and not flapping between nodes.

Learn more

Curious how clustering fits together end to end? Start with Understanding Clustering for the architecture, then follow the deployment walkthrough to stand up a two-node cluster yourself.

For questions about running Pangolin at scale, reach out to our team.