AI traffic · crawler rules
Block AI crawlers by company, or one crawler at a time
Block AI crawlers from the same screen that shows what they read. Keep training crawlers out and stay in AI answers, or keep a company out entirely, and see every refusal.
To block AI crawlers is to refuse the bots AI companies send, such as GPTBot or ClaudeBot, before your server hands them a page. Skald's crawler rules do it through nginx, the web server in front of your site, by the name each crawler sends. They leave robots.txt, the file where a site asks crawlers to stay away, open to read. Crawlers your server cannot see get a robots.txt line instead.
30-day free trial · no card · we set it up within one working day · free to self-host
GPTBotGET /pricing/403 · refusedGPTBotGET /robots.txt200 · rules stay readableChatGPT-UserGET /docs/setup/200 · a person askeda personGET /docs/setup/200 · never touchedClaudeBotGET /about/403 · refused
- served
- 3
- refused (403)
- 2
Sample requests at nginx's door. A refused crawler gets a 403 before any page is made, except robots.txt, which is always answered. People's browsers carry no crawler name, so the rule never sees them.
[01] one switch per company, one per crawler
Stay out of AI training. Stay in AI answers.
Each AI company gets one switch: Allow, Block training only, or Block. Each of its crawlers can be set on its own, and a crawler's own setting beats its company's. Flip anything and the lamps show what your server would do. The rules sit beside the crawlers they apply to, in AI traffic tracking.
OpenAI
- GPTBotcollects for trainingBlocked
- OAI-SearchBotindexes for AI searchAllowed
- ChatGPT-Userfetches for someone's questionAllowed
Anthropic
- ClaudeBotcollects for trainingBlocked
- anthropic-aicollects for trainingBlocked
- Claude-SearchBotindexes for AI searchBlocked
- Claude-Userfetches for someone's questionAllowed
- Claude-Webfetches for someone's questionBlocked
Apple
- Applebot-Extendedcollects for training · robots.txt onlyAsk in robots.txt
what nginx refuses
5 names refused with a 403, 1 asked in robots.txt
Anthropic is blocked, but Claude-User is set to Always allow on its own, so its lamp stays green. Applebot-Extended never sends a request, so only a robots.txt line can ask.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Claude-Web Disallow: /
Sample settings for three of the companies on Skald's list, with every crawler each one has there. The tiers, the override and the robots.txt suggestion are the code's own (bot_policy.py). Skald never writes your robots.txt: it suggests the lines, you add them.
[02] did they listen?
Your robots.txt says no. Did the crawlers listen?
robots.txt is a request. Your log shows what each crawler fetched anyway. Skald reads your robots.txt at most once an hour and judges every AI and search crawler's fetches with Python's own robots.txt reader, by the crawler's name. Edit the file below, or pick a common mistake, and the verdicts change as you type.
| crawler | path | times | read under | verdict |
|---|---|---|---|---|
| ClaudeBot | /blog/ | 140 | * | allowed |
| ChatGPT-User | /docs/setup/ | 96 | * | allowed |
| Bytespider | /docs/ | 70 | Bytespider | against your rules |
| OAI-SearchBot | /docs/public/setup/ | 22 | * | allowed |
| ClaudeBot | /search/ | 12 | * | against your rules |
| Googlebot | /guide.pdf | 8 | * | allowed |
| PerplexityBot | /admin/ | 3 | * | against your rules |
Against your rules, 30 days: 85 requests by 3 crawlers.
Bytespider has its own Disallow: / and fetched /docs/ 70 times anyway. ClaudeBot and PerplexityBot fall under * and took pages it closes.
Sample fetches. Skald keeps each crawler's paths per day and checks them with urllib.robotparser and the crawler's own name (robots.py). Past days are judged by the robots.txt it read last, and the tab says so. A crawler that ignores you shows here; a server rule is the way to stop it.
[03] tested before it goes live
A broken nginx file never goes live. The old one goes back.
You press Save. The dashboard only writes your choices to a file. A separate job on the server sees the change, writes the nginx map (the list of names nginx checks each request against), runs nginx -t, and reloads only when the test passes. When it fails, the previous file is put back and the tab tells you why.
- Admins only. The rules apply to every site on the server, so only an install admin may change them.
- robots.txt stays open. A blocked crawler can still read your rules.
- Nothing else is touched. The rule sits in your site's
location /, the nginx block that serves its pages; other locations never see it.
# the root job sees bot-policy.json change $ python3 -I bot_policy.py apply --policy bot-policy.json --conf /etc/nginx/conf.d/skald-bot-policy.conf --status bot-policy-applied.json # skald-bot-policy.conf.prev the file before # skald-bot-policy.conf the new map, 5 names $ nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful $ systemctl reload nginx # bot-policy-applied.json, in part {"ok": true, "changed": true, "error": null, "blocked": ["GPTBot", "ClaudeBot", "anthropic-ai", "Claude-SearchBot", "Applebot-Extended", "Claude-Web"]}
# the root job sees bot-policy.json change $ python3 -I bot_policy.py apply --policy bot-policy.json --conf /etc/nginx/conf.d/skald-bot-policy.conf --status bot-policy-applied.json $ nginx -t nginx: [emerg] unexpected "}" in /etc/nginx/sites-enabled/example.com:41 nginx: configuration file /etc/nginx/nginx.conf test failed # someone else's edit broke the config. The previous skald-bot-policy.conf goes back. No reload. # bot-policy-applied.json, in part {"ok": false, "changed": false, "error": "nginx -t refused it; previous file kept: ..."}
# the root job sees bot-policy.json change $ python3 -I bot_policy.py apply --policy bot-policy.json --conf /etc/nginx/conf.d/skald-bot-policy.conf --status bot-policy-applied.json # the new map is the file already in place: nothing to test, nothing to reload # bot-policy-applied.json, in part {"ok": true, "changed": false, "error": null, "blocked": ["GPTBot", "ClaudeBot", "anthropic-ai", "Claude-SearchBot", "Applebot-Extended", "Claude-Web"]}
Live on the site: 6 crawlers blocked.
The steps and the outcome file are bot_policy.apply's own; the paths are a sample. While the job runs, the tab says "Saved. Being applied to the site…".
[04] before you block
Block everything, and the assistants stop reading you mid-question.
Blocking a training crawler keeps your pages out of what it collects to train on, while AI search and the fetchers still read you. Blocking a fetcher means an assistant cannot read your page while someone asks about you. Pick a rule and watch the month's requests go dark, tower by tower.
- 4,820GPTBot
- 3,960ClaudeBot
- 2,100meta-externalagent
- 1,850Bytespider
- 910Amazonbot
- 640CCBot
- 1,310OAI-SearchBot
- 1,120PerplexityBot
- 600Claude-SearchBot
- 1,240ChatGPT-User
- 520Perplexity-User
- 310Claude-User
- still served
- 5,100
- refused (403)
- 14,280
- of those, a person asked
- 0
The training towers go dark. AI search indexes and the fetchers that answer a person stay lit: 5,100 requests still served.
A sample month, 12 crawlers, one rule for every company. A refused request is still a line in your log: Skald counts each crawler's 403s and shows them beside its setting, "Refused (403), 30 days". People are never touched: a visit from a ChatGPT link is a person's browser. Who AI answers send you and what each crawler covers help you choose.
[05] what a server cannot see
Google-Extended never knocks. Only robots.txt can ask it.
Some names are switches in robots.txt, not crawlers. Google-Extended tells Google whether it may train on what Googlebot fetched; Applebot-Extended does the same for Apple. No request ever carries them, so no server rule can refuse them. When your choice blocks one, its lamp says "Ask in robots.txt" and the suggested lines include it.
What it does not do
- It blocks by the name a crawler sends in each request, its user agent. That name is self-reported: the big crawlers say who they are, a scraper behind a browser name gets through.
- It works through nginx on the server Skald runs beside, once the server admin has set it up. Elsewhere the tab still shows the suggested robots.txt lines.
- A robots.txt line is a request, not a refusal. Skald suggests the lines; it never writes your robots.txt.
- It allows or refuses. It does not charge crawlers for access.
- It gives no protection from attacks. For who probes your site, see the security view.
questions
Block AI crawlers FAQ
Straight answers, including what Skald does not do.
How do I block AI crawlers like GPTBot?
Two ways. A robots.txt line asks a crawler to stay away; it is a request, and a crawler may ignore it. A rule in your web server refuses it before the page is served. Skald's crawler rules do the second through nginx, per AI company or per crawler, and suggest the robots.txt lines for the crawlers a server cannot see.
Can I block AI training but stay in ChatGPT and Claude answers?
Yes. Block training only refuses the crawlers that collect for training, such as GPTBot and ClaudeBot. The crawlers that index for AI search and the fetchers that answer a person's question, such as OAI-SearchBot, ChatGPT-User and Claude-User, still reach the site.
What does a blocked crawler see?
A refusal (403) before the page is served. Your robots.txt stays readable, so the crawler can still read your rules. Beside each crawler, Skald shows how often it was refused in the last 30 days.
Why is Google-Extended robots.txt only?
Google-Extended is a robots.txt token, not a crawler: no request ever carries it. It tells Google whether it may train on what Googlebot fetched. A server rule cannot see it, so only a robots.txt line works. Applebot-Extended is the same.
Does blocking stop every AI crawler?
No. A rule works by the name a crawler sends, and a name is self-reported. The big crawlers say who they are; a scraper that hides behind a browser name is not stopped. Skald also never charges crawlers for access: it allows or refuses.
Who can change crawler rules?
Only an install admin, because the rules apply to every site on that server. They work where Skald runs beside your nginx and the server admin has set them up. Elsewhere, the tab still shows the suggested robots.txt lines.
$ tail -f access.log | skald
See what AI does with your site. Free for 30 days.
No card. We set up hosted Skald for you within one working day, or you self-host it free. Every line of your log, sorted and added up.