AI traffic · crawlers
AI crawler tracking: who read your site, and is it real?
AI crawler tracking from the log your server already writes. Every crawler by name, checked against its owner's published addresses, with the pages it read and what it cost your server.
AI crawler tracking is counting the bots AI companies send to read your website, such as GPTBot, ClaudeBot and PerplexityBot: which pages each one fetched, how often, what it cost your server, and whether it is really who it says it is. Skald reads it from your server's access log, because a crawler never runs a script tag.
30-day free trial · no card · we set it up within one working day · free to self-host
06:05:12 192.0.2.200 "GET /.env" "GPTBot/1.2"
GPTBot's name, from an address on none of OpenAI's lists. It gets a sticker and drops to the bots tray.
OpenAI
Anthropic
Perplexity
Meta
Common Crawl
bots tray · claims to be
trainingsearch indexlive fetch, a person waits
Sample lines. Each badge is the name a crawler sends, filed under the company it names. One its owner's lists disprove drops to the bots tray.
[01] every crawler by name
Thirty-nine AI names, three reasons to come. Each one filed by why it came.
One AI company sends several crawlers, each for its own reason. Skald names every one the way it names itself and files it by why it came: to train a model, to build AI search, or to fetch a page while a person waits. Part of AI traffic tracking.
Gp1
GPTBot
- Owner
- OpenAI
- Why it comes
- Reads pages to train a model. Nobody is waiting on it.
- How its address is checked
- Against OpenAI's three published address lists.
- With "Block training"
- Refused by the nginx rule Skald writes.
The names are Skald's own crawler list, the one its blocking rules are built from. nginx is the web server in front of many sites; Skald writes the rule it reads. A name not on the list yet is filed by the general rules until it is added.
[02] trust the names
Anyone can type "GPTBot". Spot the one who did.
Scanners wear crawler names to look harmless, and every fake inflates your AI figures. Skald checks each claim the way its owner says to, and calls a request a fake only with proof. Pick a round, then pick the impostor.
Four requests say they are OpenAI's. OpenAI publishes the addresses of all three of its crawlers. Which one is lying?
Verified On OpenAI's GPTBot list.
Spoofed On none of OpenAI's three lists, and all three are fresh. Filed under bots as "claims to be GPTBot".
Verified On OpenAI's ChatGPT-User list.
Verified On OpenAI's search crawler list.
#2 is the fake. Skald files it under bots, so your AI figures stay clean.
Sample addresses. "DNS both ways" means Skald asks which name an address belongs to, then checks that name points back to the same address. The owners' lists are fetched once a day and the last good copy kept; DNS answers are kept 7 days. A list older than 7 days can no longer call anything a fake. Nothing touches the network while Skald counts.
[03] what crawling costs you
A crawler uses your server. Here is its meter reading.
Per crawler and per day: every request, the bytes you sent, the mean and 90th-percentile answer time, its errors and how often it fetched the same path again. Read from the size and time fields of your own log lines. A field your log never wrote says "not recorded", never 0.
GPTBot · yesterday · sample
log_format: … $body_bytes_sent … $request_timerequests
41
every one counted
bytes sent
2.1 MB
41 of 41 lines sized
not recorded
mean answer time
118 ms
41 of 41 lines timed
not recorded
p90 answer time
240 ms
nearest rank, one day
not recorded
distinct paths
23
no query string
re-crawl
1.8×
41 requests ÷ 23 paths
Every field is in the log, so every meter has a reading.
Sample day. p90 is the smallest time that 90% of its timed lines took no longer than. It shows for a single day, because a 90th percentile cannot be added across days.
Open any crawler for its own record
Requests per day, the pages it fetched, its status mix, the share that came from its owner's addresses, the countries of its requests, the user agents it sent and when it was last seen. Pages, countries and agents are kept for 30 days. No address is ever kept. In the sample, GPTBot asked for /llms.txt 19 times and /llms-full.txt 16 times in a month.
[04] when a crawler stops
GPTBot stopped coming last week. Would you have noticed?
A changed robots.txt or a new CDN rule can shut an AI crawler out without anyone seeing. Skald tells you when an AI or search crawler that came often has gone quiet, in plain words, judged on checked days only.
- Often first. It must have come on at least 7 of the 28 days before. A crawler that drops by twice a month can be quiet for weeks, and that is normal.
- Whole days. 7 days with no request at all, ending with the last whole day.
- Checked days. If any of the 35 days is missing or unchecked, the rule waits instead of guessing.
07:30Saturday 24 October
Skaldsite alert · medium
GPTBot has made no request for 7 days
GPTBot (AI crawler) made 1,104 requests on 24 of the 28 days 2026-09-19 to 2026-10-16, the last on 2026-10-16; none in the 7 whole days 2026-10-17 to 2026-10-23. Requests, not visits: a crawler is never a person. If it should still come, check robots.txt and any …
No alert
Sample. The alert's own wording, cut short as a phone does. Alerts show on the dashboard and go to the site's alert mail and your webhooks, Slack included. New crawlers and sudden crawler bursts open alerts too.
[05] not every machine is AI
A Slack preview reads for nobody. A CUBOT phone belongs to a person.
Your log also holds link previews, uptime monitors, SEO tools and scripts, and some real phones with "bot" in their name. Skald names those too, so a machine never passes for a person and a person is never thrown out. Guess each one, then flip it.
0 of 8 flipped
flip a cardReal rules from Skald's classifier, on sample user agents.
[06] what it cannot see
Where AI crawler tracking stops
Every crawler figure is one Skald can prove from the log. These are its edges. To refuse a crawler instead of counting it, see block AI crawlers.
What it does not do
- A user agent is self-reported. A scraper that hides behind a browser name and loads pages like a reader cannot be told apart.
- Without a server or CDN log (a script tag only, or hosts like Shopify and Webflow), AI crawlers are not seen at all. A CDN, such as Cloudflare, is a service that serves your pages from near each reader.
- Some crawlers cannot be checked: their owner publishes no address list, or Skald has no check for that name yet. They keep their name and are never called fakes.
- Google-Extended is only a robots.txt token (robots.txt is the file where a site asks crawlers to stay away). No request ever carries it, so there is no separate figure for Google's AI use of your pages.
- A new AI crawler is on nobody's list on its first day. Until it is named, it lands in bots (or people, if its name looks like a browser).
questions
AI crawler tracking FAQ
Straight answers, including what Skald does not do.
How do I see AI crawlers on my website?
Read your server's access log. AI crawlers fetch pages without running JavaScript, so Google Analytics and other script tags never see them. Skald reads the log your server already writes and lists every AI crawler by name, with the pages it read, how often, and what it cost your server.
Which AI crawlers does Skald recognise?
GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI; ClaudeBot, Claude-SearchBot and Claude-User from Anthropic; PerplexityBot and Perplexity-User; meta-externalagent, Amazonbot, Bytespider, CCBot, Applebot-Extended and others. Each is named the way it names itself, and filed by why it came.
How does Skald know a crawler is really GPTBot?
Anyone can write GPTBot in a request. Skald checks the address against the list OpenAI publishes, and does the same for Anthropic, Perplexity, Google, Bing, Apple, DuckDuckGo and Meta, by their lists, DNS or network. A request that uses GPTBot's name from an address OpenAI does not use shows as "claims to be GPTBot", under bots.
What does "not verifiable" mean?
There was nothing to check the address against: the owner publishes no list, Skald has no check for that name yet, or the list could not be fetched. The crawler keeps its name and is marked not verifiable. Skald never calls a crawler a fake without proof.
Can I see what an AI crawler costs my server?
Yes. Per crawler, Skald shows requests, bytes sent, mean answer time, p90 time for a single day, errors, distinct paths and re-crawls. If your log does not record sizes or times, those figures say not recorded, never 0.
Can Skald block an AI crawler?
Skald can apply crawler rules through nginx on the server it runs beside, by the name a crawler sends. Some crawlers are robots.txt only. How blocking AI crawlers works.
$ tail -f access.log | skald
See what AI does with your site. Free for 30 days.
No card. We set up hosted Skald for you within one working day, or you self-host it free. Every line of your log, sorted and added up.