Help:User journeys/Combatting bots and scrapers

Journey info · Platform: Docker Compose & Kubernetes · Time: ~20 minutes

This is a guide to reducing the load that bots, scrapers, and AI/LLM crawlers can put on a Canasta wiki, using built-in tools such as the CrawlerProtection extension, CrowdSec and Caddy. These are not the only tools you can use; Cloudflare, for example, is an extremely popular tool that you can use outside of Canasta. See the guide Handling web crawlers on mediawiki.org for a more comprehensive listing of tools and approaches that can be used with MediaWiki.

For information on enabling CrowdSec within Canasta, see Help:CrowdSec. For additional information on how Caddy works within Canasta, see Help:Networking and TLS. Most of the measures here work on both orchestrators — Docker Compose and Kubernetes: the CrawlerProtection settings, the Caddy User-Agent block in config/Caddyfile.site, and CrowdSec are applied the same way on each. The one exception is the robots.txt step, which uses a Compose-only mount; the note there covers Kubernetes.

Types of crawlers

There are, roughly speaking, four types of crawlers that can hit your site. In generally decreasing order of desirability, they are:

Search-engine crawlers (e.g. Googlebot, Bingbot)
AI search/answer crawlers (e.g. OAI-SearchBot, PerplexityBot)
AI training crawlers (e.g. GPTBot, ClaudeBot, CCBot)
Abusive scrapers that ignore robots.txt or hammer the site, sometimes even maliciously.

Ideally you can take an approach that blocks the last one or two types while allowing the others access.

Blocking expensive pages with CrawlerProtection

The most effective measure against scrapers is to stop serving them expensive pages at all. Canasta includes the CrawlerProtection extension, which denies anonymous access to resource-intensive action URLs (e.g. ?action=history) and special pages (page histories, Special:WhatLinksHere, Special:RecentChanges, …), while leaving normal article reads and logged-in editors untouched.

You can enable and configure it in either a global settings file under config/settings/global/ (applied to every wiki on the instance) or a per-wiki file under config/settings/wikis/{wiki-id}/ (just that wiki).

Here is a reasonable configuration for CrawlerProtection:

wfLoadExtension( 'CrawlerProtection' );

// Deny anonymous requests with a raw 403 (skips MediaWiki rendering — faster,
// and sheds the most load).
$wgCrawlerProtectionRawDenial = true;
$wgCrawlerProtectionRawDenialText =
	"You must be logged in to view this page." .
	"<br><button onclick=\"history.back()\">Go Back</button>";

// Resource-intensive special pages to deny to anonymous users.
$wgCrawlerProtectedSpecialPages = [
	'mobilediff',
	'recentchangeslinked',
	'recentchanges',
	'relatedchanges',
	'whatlinkshere',
	'specialpages',
	'browse',
	'browsedata',
	'random',
];

// Action URLs to deny (page history is the big one).
$wgCrawlerProtectedActions = [ 'history' ];

No restart is needed for these changes (though if you are using GitOps, you should commit them so they are tracked).

You can also exempt trusted automation, such as monitoring and your own bots, from these restrictions with $wgCrawlerProtectionAllowedIPs.

Behavioral detection with CrowdSec

If CrowdSec is enabled, your engine already bans certain scraping behavior, regardless of IP address. The bundled crowdsecurity/caddy collection includes base-http-scenarios, which contains:

http-crawl-non_statics — aggressive crawling of dynamic pages (bulk content scraping), and
http-bad-user-agent — requests from known scraper/bot user-agents.

Legitimate crawlers (such as Googlebot and Bingbot) are protected from false bans by the bundled seo-bots-whitelist.

You can confirm the scenarios are active with:

canasta crowdsec scenarios

There is nothing to configure here — it is the default.

Blocking IP addresses with CrowdSec

Blocklists pre-emptively block IPs by reputation. In the CrowdSec Console under "Blocklists", subscribe this engine to free lists that catch scraper infrastructure:

Free Proxies — scrapers commonly route through open proxies.
Tor exit nodes — if you do not expect legitimate Tor readers.

You already receive the CrowdSec Community Blocklist via the Central API. CrowdSec's dedicated AI Crawlers blocklist is a paid list, but CrowdSec offers it free to open-source community projects. A public wiki usually qualifies; email community@crowdsec.net to request access. See Help:CrowdSec#Community blocklist for how blocklists reach the engine and how to read canasta crowdsec status.

Block self-identifying AI bots at the edge with Caddy

Well-behaved AI crawlers announce themselves by User-Agent and are not considered "malicious", so CrowdSec's blocklists will not stop them. You can nevertheless block them at the edge in config/Caddyfile.site in the instance directory (it is imported into the site block, and a matched handle short-circuits before MediaWiki):

@ai_scrapers header_regexp User-Agent (?i)(GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|anthropic-ai|Claude-Web|CCBot|PerplexityBot|Perplexity-User|Bytespider|Amazonbot|Meta-ExternalAgent|FacebookBot|Diffbot|ImagesiftBot|Omgilibot|cohere-ai|YouBot|DataForSeoBot|Timpibot)
handle @ai_scrapers {
    respond "Automated AI/LLM scraping of this wiki is not permitted." 403
}

Apply it with:

canasta restart

Two things to get right:

Do not list Google-Extended or Applebot-Extended here — those are robots.txt opt-out tokens, not real user-agents, and never appear in a request.
To stay citeable in AI search results, drop the search crawlers (e.g., OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User) from the list.

The crawler list is always evolving; the community-maintained ai.robots.txt project is a good source to keep it current.

Collapse cache-busting index.php requests with Caddy

Scrapers that want to defeat your caching do not need to request expensive pages. They only need to request uncacheable ones.

/w/index.php renders the wiki's main page whenever no title (or other page identifier) is supplied — whatever else the query string contains. Because each distinct query string is a distinct URL, none of them share a cache entry: every one is a miss in Varnish and a full render at the origin. A scraper appending an incrementing or random parameter therefore gets a fresh, full-cost page render every time, and the parameter can be called anything at all.

Measured on a wiki with Varnish in front, the same main page served two ways:

/wiki/Main_Page — cache hit, 6–8 ms
/w/index.php?zzz=<random> — never cacheable, 87–94 ms

Identical content, roughly 14× the cost.

The CrawlerProtection setting $wgCrawlerProtectedQueryParams (see above) denies this shape for parameter names you list, and is worth setting when you can see a specific name in your logs. It cannot close the general case, though: the name is incidental to the cost, so a scraper that renames its parameter — or simply nests it inside another parameter's value — is through again. Blocking on the absence of any page identifier does not have that weakness, and Caddy can do it before the request ever reaches Varnish or MediaWiki.

The redirect

Add to config/Caddyfile.site in the instance directory. Use this form for a single wiki, or a farm whose wikis are on separate domains or subdomains:

@titleless {
	path /w/index.php
	not query title=*
	not query curid=*
	not query oldid=*
	not query diff=*
	not query search=*
	not query action=*
}
redir @titleless / 301

If your farm has path-based wikis (for example a second wiki at /draft/), use this instead — it captures the wiki's path prefix so each wiki redirects to its own root:

@titleless {
	path_regexp wiki ^(/[^/]+)?/w/index\.php$
	not query title=*
	not query curid=*
	not query oldid=*
	not query diff=*
	not query search=*
	not query action=*
}
redir @titleless {re.wiki.1}/ 301

Apply it with:

canasta restart

A request carrying none of those six parameters is redirected to the wiki root, which is a single cacheable URL. Requests that name a page, a revision, a search or an action are passed through untouched.

Before you enable it

Check that nothing on your wiki legitimately requests index.php without one of the six parameters. This lists successful requests that the redirect would have caught, on every wiki in the instance:

canasta maintenance exec -- grep -h index.php /var/log/apache2/access_log_$(date +%Y%m%d) \
  | awk '$9==200' \
  | grep -oE "GET [^ ]*/w/index\.php\?[^ ]*" \
  | grep -vE "[?&](title|curid|oldid|diff|search|action)=" \
  | sort | uniq -c | sort -rn | head -20

The pipeline above runs in your own shell rather than inside the container. canasta maintenance exec passes a pipeline to sh -c correctly if you prefer to run it there, but escaping $ through nested shells is easy to get wrong — keeping the pipeline outside avoids the problem.

Expect scraper-shaped noise. Do not judge the output by parameter name alone. Names borrowed from real features are common — requests carrying bookcmd, collection_id and writer look like Special:Book traffic, but the genuine article always carries title=Special:Book and so is never caught by the rule. Two signals separate them reliably:

canasta maintenance exec -- grep -h index.php /var/log/apache2/access_log_$(date +%Y%m%d) \
  | awk '$9==200' | grep -vE "[?&](title|curid|oldid|diff|search|action)=" \
  | awk -F'"' '{print $4}' | sort | uniq -c | sort -rn | head

That prints the Referer of each candidate. Real users reach these URLs from a page on your wiki; scrapers arrive with "-". Substituting $6 for $4 prints the User-Agent instead, which usually shows a single forged browser string dominating. If every candidate has no referer, the rule is safe to enable.

If a genuine feature does appear, add its identifying parameter to the matcher as another not query line before enabling the redirect.

Things to get right

Do not wrap the redir in a handle block. Inside handle, redir / 301 is parsed with / as an inline path matcher rather than as the destination, and the result is a silent empty 200. Unlike respond, redir is ordered before reverse_proxy and short-circuits correctly on its own. (If you must use handle, write redir * / 301.)
config/Caddyfile.site is bind-mounted into the container as a single file. Editors that write a new file and rename it over the original change the file's inode, and the container keeps reading the original one — so reloading Caddy in place reports success while serving stale configuration. Applying the change with canasta restart avoids this entirely. To confirm which file the container is reading, compare inodes — they should match:

stat -c %i config/Caddyfile.site
canasta maintenance exec -s caddy -- stat -c %i /etc/caddy/Caddyfile.site

The redirect target is the wiki root, not a hard-coded main page title. This keeps the rule correct on wikis whose main page is not called Main_Page, and on farms whose wikis have different main pages. The root is itself a redirect to the canonical main page on most instances, so a followed request makes two cheap hops rather than one.
The rule matches on the script path (/w/index.php), never on article paths, so it is unaffected by $wgArticlePath. It works unchanged with CANASTA_ENABLE_VERY_SHORT_URLS=true.
Very short URLs and path-based wikis are mutually exclusive in Canasta, so use the simple form on any instance with very short URLs enabled.
Adapting this outside Canasta: the matcher assumes $wgScriptPath is /w. On a wiki serving index.php from the document root, match /index.php instead — do not try to match both in one regular expression, as /w/index.php would then be read as a path-based wiki called w.

Verifying

Requests that should now be redirected:

curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" "https://example.org/w/index.php?zzz=1234567"

Expect 301 to the wiki root. Then confirm that ordinary traffic is untouched — every one of these should still return 200 (or a normal MediaWiki redirect):

for u in "/w/index.php?title=Main_Page&uselang=fr" \
         "/w/index.php?title=Special:UserLogin&returnto=Foo" \
         "/w/index.php?oldid=1" \
         "/w/index.php?search=test" \
         "/wiki/Main_Page?fbclid=abc" \
         "/w/load.php?modules=startup" \
         "/w/api.php?action=query&format=json"; do
  curl -s -o /dev/null -w "%{http_code}  $u\n" "https://example.org$u"
done

The fbclid case matters: link-tracking parameters such as fbclid and utm_source are appended automatically by social and email platforms, and arrive on ordinary article URLs from real readers. Because the rule only ever inspects /w/index.php, those requests are never examined.

Using robots.txt

robots.txt asks compliant crawlers to stay out, and is the only way to express the Google/Apple AI-training opt-out tokens. Canasta serves /robots.txt from robots.php (it disallows Special:, MediaWiki: and /w/, and advertises sitemaps). Do not serve a competing /robots.txt from Caddy, which would override it.

Docker Compose only: the procedure below appends to robots.txt through a bind mount in docker-compose.override.yml, which has no Kubernetes equivalent. On Kubernetes there is no first-class way to inject extra-robots.txt, so rely on the other layers above — CrawlerProtection and the Caddy User-Agent block both work on Kubernetes — for the practical protection; only the Google/Apple AI-training opt-out tokens, which can only be expressed in robots.txt, are unavailable there.

robots.php appends /var/www/mediawiki/extra-robots.txt to its output when that file exists. That path is not bind-mounted by default, so add a persistent mount via docker-compose.override.yml. Keep the source file under config/ so backups and GitOps capture it automatically (everything under config/ is tracked).

Create the file first (it must exist before up, or Docker creates a directory):

cat > config/extra-robots.txt <<'EOF'
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
EOF

Add the mount (Compose merges this onto the base web volumes, preserving the existing mounts):

services:
  web:
    volumes:
      - ./config/extra-robots.txt:/var/www/mediawiki/extra-robots.txt:ro

Apply it with:

canasta restart

Google-Extended and Applebot-Extended belong here, not in the Caddy block, because they opt your content out of Google/Apple AI training while still allowing those companies' search crawlers to index you. Note that Bytespider has been observed ignoring robots.txt, which is why it also appears in the Layer 4 hard block.

A note on backup and GitOps: config/extra-robots.txt is captured by canasta backup (restic) and tracked by GitOps (it lives under config/). The docker-compose.override.yml that mounts it is also backed up by restic and is not ignored by GitOps (only docker-compose.override.yml.example is) — on a GitOps instance, track it once with canasta gitops add docker-compose.override.yml and it rides along thereafter. So the whole setup survives a restore or a GitOps pull. See Help:Backup and restore.

Validation

To confirm that CrawlerProtection denies an anonymous history request, but not normal article reads, you can simply visit the pages, or you can call the following:

curl -sI "https://YOUR-WIKI/w/index.php?title=Main_Page&action=history" | head -1  # expect 403
curl -sI "https://YOUR-WIKI/" | head -1                                            # expect 200

To confirm that the edge block returns 403 for a blocked user-agent but serves a normal browser:

curl -sI -A "GPTBot" https://YOUR-WIKI/ | head -1       # expect 403
curl -sI -A "Mozilla/5.0" https://YOUR-WIKI/ | head -1  # expect 200

To confirm that your robots.txt file is being served correctly, simply visit the file https://YOUR-WIKI/robots.txt .

Confirm CrowdSec is enforcing the community and any console blocklists:

canasta crowdsec status

To confirm that the index.php redirect collapses a cache-busting request while leaving ordinary traffic alone, visit these three URLs in a browser:

https://YOUR-WIKI/w/index.php?zzz=1234567 — the address bar should jump to your wiki's root and the main page should load. Make up a different number each time you test, so a cached redirect cannot give you a stale result.
https://YOUR-WIKI/w/index.php?title=Main_Page&uselang=fr — the main page should load in French, with the query string still in the address bar.
Any article on your wiki with ?fbclid=abc appended (/wiki/Some_Page?fbclid=abc, or /Some_Page?fbclid=abc if very short URLs are enabled) — the article should load normally, query string intact.

The third check is the important one: link-tracking parameters such as fbclid and utm_source are appended automatically by social and email platforms and arrive on ordinary article URLs from real readers, so confirming they are untouched is what tells you the rule is scoped correctly.

If you prefer to check from the command line:

curl -sI "https://YOUR-WIKI/w/index.php?zzz=1234567" | head -1                 # expect 301
curl -sI "https://YOUR-WIKI/w/index.php?title=Main_Page&uselang=fr" | head -1  # expect 200
curl -sI "https://YOUR-WIKI/ARTICLE?fbclid=abc" | head -1                      # expect 200

Undoing these protections

To disable CrawlerProtection, remove (or comment out) the CrawlerProtection block from your settings file (config/settings/global/ or config/settings/wikis/{wiki-id}/).
To undo Caddy blocking, remove the @ai_scrapers block from config/Caddyfile.site.
To undo the index.php redirect, remove the @titleless matcher and its redir line from config/Caddyfile.site.
See Help:CrowdSec for how to disable CrowdSec. To disable it for only a certain set of trusted IPs, see Help:CrowdSec#Whitelisting trusted IPs.
To undo your robots.txt changes, remove the extra-robots.txt volume line from docker-compose.override.yml (delete the file too if unused). If the override file is now empty, you can remove it entirely.

After any of these changes, you will need to call canasta restart.

Production considerations

Tune, do not carpet-block. Keep search-engine and (optionally) AI-search crawlers allowed so your wiki(s) stay discoverable and citeable; block training crawlers and abusive scrapers. Revisit the User-Agent list periodically against ai.robots.txt.
Wiki farms. The Caddy block applies to the whole site address; on a farm it covers every wiki on that hostname. CrawlerProtection applies to every wiki when placed in config/settings/global/, or to a single wiki when placed in config/settings/wikis/{wiki-id}/.

Troubleshooting

A page anonymous users need returns 403. CrawlerProtection is denying a special page or action they legitimately use — remove it from $wgCrawlerProtectedSpecialPages / $wgCrawlerProtectedActions, or exempt the source IP via $wgCrawlerProtectionAllowedIPs (a config edit takes effect immediately — no restart is needed).
Caddy block does nothing. The directive must be a handle @ai_scrapers { ... } block in config/Caddyfile.site — a bare respond is ordered after reverse_proxy and never fires. Re-apply with canasta restart and re-run the curl -A test.
robots.txt lines missing. Confirm the bind mount resolved to a file, not a directory: docker compose exec web ls -l /var/www/mediawiki/extra-robots.txt. If it is a directory, you created the mount before the host file existed — remove it, create extra-robots.txt, and canasta restart.
A blocked AI bot still appears in logs. Some crawlers (e.g. Bytespider) spoof or ignore controls; CrowdSec's http-bad-user-agent and the community blocklist catch many of these by behavior and reputation.

The index.php redirect does nothing after editing Caddyfile.site. The file is bind-mounted into the container as a single file, and most editors save by writing a new file and renaming it over the original — which changes the inode, leaving the container reading the file it was started with. caddy reload then reports success while serving the old configuration. Compare the inodes to confirm — they should match: stat -c %i config/Caddyfile.site and canasta maintenance exec -s caddy -- stat -c %i /etc/caddy/Caddyfile.site. A canasta restart re-establishes the mount and avoids the problem entirely.
The index.php redirect returns an empty 200 instead of a redirect. The redir has been wrapped in a handle block, where redir / 301 is parsed with / as an inline path matcher rather than as the destination. Unwrap it — redir is ordered before reverse_proxy and short-circuits without handle — or write redir * / 301 if the block is needed for other reasons.
A page that used to work now redirects to the wiki root. Something requests index.php with a parameter that identifies a page but is not one of the six in the matcher — most likely an extension with its own entry point. Add that parameter as another not query line. The pre-enable log check in Before you enable it finds these before they bite.
The redirects do not appear in the wiki's access log. They will not. Caddy answers them at the edge, so the request never reaches Varnish, Apache or MediaWiki — which is the point. Confirm the rule is working from the client side instead, using the checks in Validation. A falling count of title-less index.php requests in the access log is the other signal that it is taking effect.