How to Block Bots With CrowdSec and Pangolin
Anything you put on the public internet gets crawled, scraped, and probed by automated traffic within minutes. Login pages get credential stuffing, apps get scraped, and headless browsers hammer endpoints that were only ever meant for people. IP reputation and rate limits catch some of it, but a lot of modern bot traffic comes from fresh residential IPs and moves slowly enough to look normal.
Pangolin 1.26 adds an opt-in option to the installer that turns on CrowdSec's bot detection, introduced in CrowdSec 1.8, for self-hosted installs. With it enabled, a browser has to pass a short, invisible check before it reaches any resource behind Pangolin. Real visitors usually see a brief challenge page once and then carry on. Automated clients are rejected at the proxy, before they reach your application.
Bot detection is in alpha. It is off by default and only applies to self-hosted Pangolin installs that use CrowdSec.
How It Works
Pangolin's installer already sets up CrowdSec alongside Traefik when you run it with --crowdsec. Traefik passes every request through the CrowdSec bouncer plugin, which asks CrowdSec's AppSec component whether to allow it. Bot detection is an extra set of AppSec rules on top of that.
When a request arrives without a valid challenge cookie, AppSec serves a challenge page instead of your app. The page does two things in the browser:
- Proof of work. The browser solves a small computational puzzle. It takes a moment for one visitor but adds up for anyone sending large volumes of requests.
- Fingerprinting. The page collects signals about the browser and device: automation markers like WebDriver, Selenium, Playwright, or the Chrome DevTools Protocol, plus consistency checks across the user agent, screen, GPU, timezone, and languages.
The results are sent back to CrowdSec, which scores the fingerprint. Strong automation signals, such as a detected CDP or WebDriver session, push the score over the threshold on their own. Weaker signals, like a headless screen size or a UTC timezone, add up. If the score is below the threshold, the browser gets a sealed __crowdsec_challenge cookie and goes through to your resource. By default the cookie lasts 12 hours, so visitors are not challenged on every page load. If the score is too high, the request is rejected.
CrowdSec also ships exclusion configs for verified good bots, including search engines, social link previews, and uptime monitors. A client only counts as verified if its IP is in the vendor's published ranges or passes a forward-confirmed reverse DNS check, so a scraper calling itself Googlebot is still challenged.
Before You Turn It On
Bot detection sits in front of every resource that goes through the CrowdSec middleware, not just one. That is what makes it effective, and it is also the main thing to plan around.
Anything that is not a full browser will fail the challenge. That includes:
- Mobile apps that talk to a self-hosted backend
- API clients, CLIs, and scripts using tools like
curl - Webhooks from third-party services
- Media and sync clients, such as native apps for a home media server or file sync
Enable It With the Installer
On a new install, follow the quick install guide and run the Pangolin installer with CrowdSec enabled:
./installer --crowdsec
After you confirm the CrowdSec install, the installer asks:
Would you like to enable CrowdSec bot detection (alpha)? Browsers must solve a challenge before reaching any resource; non-browser clients such as mobile apps and API tools may be blocked.
Answer yes. The installer adds the crowdsecurity/appsec-bot-challenge collection to the CrowdSec container and loads the bot detection AppSec configs alongside the default ones. Everything else in the CrowdSec setup stays the same.
Enable It on an Existing Install
If you already run CrowdSec with Pangolin, you can turn bot detection on by hand. In short:
- Use version
v1.8.1or newer of the CrowdSec bouncer plugin inconfig/traefik/traefik_config.yml. - Add
crowdsecurity/appsec-bot-challengeto theCOLLECTIONSvariable of the CrowdSec container indocker-compose.yml. - In
config/crowdsec/acquis.d/appsec.yaml, replaceappsec_configwith a list that loads the bot rules:
listen_addr: 0.0.0.0:7422
appsec_configs:
- crowdsecurity/appsec-default
- crowdsecurity/appsec-bot-*
name: myAppSecComponent
source: appsec
labels:
type: appsec
- Make sure AppSec is enabled on the
crowdsecmiddleware inconfig/traefik/dynamic_config.yml(crowdsecAppsecEnabled: true). Installer-based setups have this on already. - Restart the stack with
docker compose down && docker compose up -d.
The full steps are in the CrowdSec bot detection docs. Use a bouncer version that supports bot detection. CrowdSec warns that running it behind a bouncer without support will most likely refuse every client.
Check That It Works
Open one of your resources in a private browser window. You should see the challenge page briefly, then your app. Requests from a plain curl should get the challenge HTML back instead of your app:
curl -i https://app.example.com
To see what CrowdSec is rejecting, check the bot detection alerts and metrics:
docker exec crowdsec cscli alerts list --kind bot-detection
docker exec crowdsec cscli metrics show bot-detection
Each alert lists the signals behind the score, such as cdp or webdriver, so you can see why a client was blocked.
Where It Fits With Pangolin's Other Controls
Bot detection decides whether a visitor is a real browser. It does not decide who that visitor is or whether they should have access. Those jobs still belong to Pangolin:
- Authentication with SSO, passwords, PINs, or passkeys decides who gets in.
- Rules allow, block, or skip auth by IP, CIDR, path, or country.
- CrowdSec's IP reputation and AppSec rules still block known bad IPs and common exploit payloads, with or without bot detection.
Layered together, a request has to come from an acceptable IP and location, pass the bot check, and then sign in before it reaches your app.
FAQ
Do visitors have to click anything?
No. The challenge runs in the background. Visitors see a short interstitial page, usually for a second or two, and then the page loads. After that they are not challenged again until the cookie expires, which is 12 hours by default.
Will this block search engines?
Not if you load CrowdSec's search engine exclusion config. It exempts crawlers whose IPs or reverse DNS match the vendor's published ranges. Crawlers that only claim to be a search engine in their user agent are still challenged.
Why is my mobile app or API client being blocked?
Bot detection only lets through clients that can run JavaScript and pass the browser fingerprint check. Native apps, scripts, and API tools cannot. Exclude the paths they use, allowlist their IPs, or turn bot detection off for that setup.
Is this available on Pangolin Cloud?
This option is for self-hosted installs that use CrowdSec through the Pangolin installer.
How do I turn it off?
Remove crowdsecurity/appsec-bot-* from appsec_configs in config/crowdsec/acquis.d/appsec.yaml and restart CrowdSec.
Is Pangolin open source?
Yes. Pangolin is open source and self-hostable. You can deploy the whole platform yourself, use Pangolin Cloud, or mix hosted and self-hosted components.
See Also
- CrowdSec with Pangolin: installation, configuration, and the full bot detection steps
- Pangolin 1.26 release notes: bot detection shipped alongside session management and SSH logs
- How to geo-block with Pangolin: restrict resources by country with rules
- CrowdSec bot detection docs: how the challenge and scoring work in depth
- CrowdSec 1.8 announcement: the release that introduced WAF bot detection
About Pangolin
Pangolin is the open-source access platform that lets IT and security teams connect their workforce, AI agents, and infrastructure without the friction of legacy VPNs and stitched-together security tools. By unifying a zero-trust VPN, zero-trust reverse proxy, privileged access management, and an identity-aware AI gateway under one policy model, Pangolin helps organizations shrink their attack surface, contain stolen credentials, and pass audits with less work. Teams can roll it out one service at a time, self-hosted or on Pangolin Cloud, and verify exactly how their access is secured because every component is open source.
